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ABSTRACT 

Intended for directors of special education projects, 
the manual provides guidelines for program evaluation. It is 
explained that the manual developed out of the experiences of the 
staff of the Leadership Training Institute in Learning Disabilities 
which provided technical assistance to 43 state projects. The 
manual's eight major sections focus on the following topics: 
alternative approaches to program evaluation and accountability; 
framework involving implementation, progress, and outcome aspects of 
evaluation; initial considerations in planning an evaluation; 
possible evaluation questions and priorities of various interested 
groups and agencies; selection of issues to be evaluated; 13 types of 
evaluation designs ranging from the simple to the more complex; 
suggestions for writing the evaluation plan including casting 
evaluation questions into behaviorally stated questions; and the 
transfer of responsibility to the on-site project director. Stressed 
for successful program evaluation are cooperation with all interested 
parties (such as educational administrators) and the use of 
evaluation results as feedback for modification of the program model. 
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Preface 



During the academic years of 1971-74, the Bureau of Education for 
the Handicapped, U.S.O.E., supported a technical assistance team. This 
team provided consultation services to model centers for learning disabled 
children. During the first year it became apparent that there was a need 
for intensive technical assistance in a wide variety of program areas. Many 
of the programs needed considerable assistance in program evaluation. This 
manual is a result of the experiences of its authors who helped provide 
assistance in the program evaluation area to these child service demon- 
stration programs. 

The child service demonstration programs and the Leadership Training 
Institute technical assistance team were funded under the Learning Disability 
Act of 1969, which was incorporated into the Elementary and Secondary 
Education Act of 1970 as Title VI-G. The focus of these programs was 
service to children with learning disabilities, through the establishment 
of a model service center in each state. Designed, established, and operated 
through the offices of each State Department of Education, these model centers 
were conceived to be testing grounds for suitable programming for learning 
disabilities. This program could then be disseminated throughout the state. 
The role of the technical assistance team at the Leadership Training Institute 
was to assist each project in maximizing its potential in all areas of its 
functioning wherever possible (planning, implementation, administration, 
substcintive services, evaluation and replication). 

Technical assistance in evaluation is to be distinguished from evalua- 
tion of a program itself and from establishment of a complete evaluation 
plan for each program. The task we faced was providing advice in program 
evaluation to as many as 43 different programs — each with different 
goals and objectives. Each of these programs had staffs with vaurying degrees 
of evaluation expertise. 

Providing meaningful input to so many varied programs required that 
our evaluation model be flexible, while at the same time it also had to be 
sufficiently specific so that our assistance would have meaningful effects 
upon the behavior and competence of each program's evaluation. 

This manual describes many of the issues that we sought to have the 
programs consider. In a sense, it is a "how-to-do-it" manual, though it 
is not as specific as a cookbook. We pose the problems that a person plcinning 
an evaluation would face and we attempt to delineate the issues cind the 
relevant considerations in order to plan a coherent, comprehensive evaluation. 
This manual deals with the problems that are typically faced in the order that 
they usually occur. These problems are by no means simplistic and, hopefully, 
are not dealt with as such. Though brief, the manual attempts to treat each 
problem meaningfully in the context that these problems actually occur. This 
approach is in marked contrast to the text books in evaluation which organize 
topics according to theoretical models or the technology of the evaluation 
process . 
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The problem-solving orientation should make this manual useful to 
persons who have the task of pleuming special education service progreuns 
as well as those in charge of carrying out these programs once they are 
funded. The m^mual presupposes that persons who are in the role of initiating 
special education prograuns are typically not trained in progreun evaluation 
procedures. No technical knowledge of progreun evaluation is therefore 
assumed. This manual will not make the reader a competent progreun evaluator 
but it should allow him to maKe good judgements eJjout how the program evaluation 
process should be planned and administered. With the assistance of someone 
knowledgeable about evaluation methodology, the program initiator should 
then be eible to communicate effectively with this evaluator and in so doing 
help create a con^etent and informative progrcim evaluation. 




Introduction 



This manual is divided into nine sections. In this first section, 
our program evaluation approach and alternative views concerned with the 
scope of evaluation, the role of staff evaluation, attitudes toward evalu- 
ation, and the issue of accountedaility are described. The second section 
details the evaluation framework that we have adopted. It involves imple- 
mentation, progress, and outcome aspects of evaluation. The third section 
is concerned with initial considerations in planning an evaluation. It 
deals first with the role of the project's pre-history in constructing an 
appropriate eind workable evaluation and then with the procedures that have 
to be engaged in and the problems that have to be dealt with in order to 
construct a meaningful evaluation. 

Section four exemiines the range of possible evaluation questions and 
the priorities of various interested groups and agencies. The point is 
stressed that evaluation should be useful to as many persons as possible, 
including various interest groups, the project staff, the funding agent, 
cuid the initiator. 

The selection of issues to be evaluated is considered in section five. 
The persons who should be involved at this stage cind the role of dimensions 
such as the eveU.uation ' s purpose, audience, priorities and costs are 
discussed. 

Somewhat more technical information on evaluation designs is considered 
in section six. Our purpose here is to inform the project initiators about 
the range of question-answering "designs" that exist so that he ccin converse 
comfortably and intelligently with evaluators. 

Suggestions for writing an evaluation plan, as is needed in preparing 
grant requests, are provided in section seven. Goals are distinguished from 
objectives; casting evaluation questions into behaviorally stated objectives 
is briefly discussed. The aim of these technical sections is to provide the 
program initiator with an information base from which to converse with evalua- 
tors rather than to teach the initiator how to become an evaluator himself. 

The final sections are concerned with transferring responsibility for 
the evaluation to the on-site project director. Considerations in selecting 
an on-site evaluator are discussed briefly. A summary section concludes the 
manual. 

Conceptualizing Program Evaluation 

Program evaluation is viewed as an ongoing, dynamic activity through- 
out the life of the program - from pre- planning to post-project assessment 
cUid decision maUcing by interested observers. Evaluation is an information 
gathering procedure, utilized specifically for decision maUcing. Sometimes 
this decision-maUcing process is used for modifying aspects of the ongoing 
program; at other times the procedure is used to make decisions about program 
continuation. In all cases, information gathering must serve a specific stated 
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purpose. That is, one must always ask " What question will the data answer , 
or v^at need or demand will the data fulfill? " Only information which 
represents the most efficient, reasonable method of cmswering each question or 
fulfilling each need should then be collected. 

Viewing the evaluation process as an information gathering system for 
decision-making changes the focus from that of judging program worth at the 
conclusion of a progreun to that of monitoring cmd adjusting those aspects 
of the program about which one is most intimately concerned. Rather than 
being simply a cong)onent of the program, the evaluation process permeates 
the totaQ program because people must continually make judgements eOoout the 
correctness, appropriateness, or vatlue of their activities emd those of 
others. Evaluation methodology cmd its related tools singly represent the 
technology for both proposing cmd cmswering questions about these activities 
in a scientific, objective manner. 

As a body of techniques, evaluation methodology is one accepted method 
of gathering information which will provide a rational and logical meems of 
supporting statements about the project. Of course, evatluation methodology 
can be used ritually, data being collected without specific purpose or to 
satisfy some erroneously perceived demand. Like emy other set of tools, 
evaluation methodology is limited by the creativity emd the ingenuity of the 
user. Being based on the rather rigid strictures of the scientific method, 
evaluation methodology is already very limited in that it must trade much of 
the richness of project events for objective measurement. Consequently, the 
methodology must be used sensitively emd conscientiously if the information 
gathered and the conclusions drawn are to be useful. Evaluations \indertaken 
ritually and without enthusiasm might just as well not be done; the problem 
in these cases lies as much or more with the persons accomplishing the eval- 
uation as with evaluation methodology per se . 

The process of casting questions of concern about the project into 
measuraQsle objectives is considered later. At this point, however, it must 
be acknowledged that the science of evaluation is only one way for people to 
learn eODOut the efficacy of their own actions, about their effect on others, 
and about the organizations in which they work. The keen eye and open mind 
of a conscientious project director will certainly see many things that the 
empirically-based methods of the evaluator might missj scientific methodology 
should not be seen as competing with the good judgment of the program staff 
or of those who must decide about program modification and continuation. Both 
scientific evaluations amd the testimony of those who have been involved in 
the program itself or similar programs cem be blended to yield a comprehensive 
understemding of the program's functioning. Evaluation methodology, used 
sensitively, has the advcmtage over testimony of demonstrating the reality 
of certain assertions in cm objective manner so that a "neutral" observer 
is able to draw his own conclusions. We adopt the position that evaluation 
methodology should be as sensitive as possible to the nuances in information 
contained in programs, while at the saiae time objectifying these data so 
that a body of reliable information can be amassed for use by others. 

There obviously does not exist a single correct method for prograun 
evaluation. What specific aspects of the prograim are to be evaluated will 




be unique judgments of the people involved. The evaluation methods chosen 
will also he xiniqae depending upon resources and the conclusions to be 
drawn from the information gathered. As such, evaluation is a personal 
enterprise, having no set formula. The available tools may be standard, 
but the specific methods chosen and the manner in which the tools are used 
will necessarily be highly individualistic. 

The task of the progrcun initiator is to assure that he has sufficient 
technical capabilities at his disposal to answer questions in a logical 
emd rational manner. Program evaluation can be extremely useful in fulfilling 
this responsibility. However, it must be recognized that program evaluation 
is a specialty area in which persons receive advanced training at the 
Doctorate Level. Viewed within this context, there is no reason to assume 
that special education program initiators or directors will possess a high 
degree of competence in this area nor is there any particular reason why 
they should have such competence. Too often, the dememds for accountability 
have been placed upon persons who lack the necessary interest or proper 
training. These demands also distract them from their particular area of 
competence. Persons cast in the new role of project evaluator as a portion 
of their directorship job frequently feel uncomfortable, if not totally 
inadequate, when confronted with this difficult task. If accountability 
is to be taken seriously, both by funding agents and by the recipients of 
those funds, each must recognize that special training is necessary for 
meciningful evaluations. Consequently, we strongly advise that project 
initiators seek the assistance of a trained evaluator when they plan a 
project. 

Some Alternative Views of Evaluation 

There are at least four other ways to view evaluation. These other 
ways are either inconsistent with our viewpoint or represent a subset of 
the evaluation issues that will be discussed. 

Program Evaluation versus Outcome Evaluation 

The first takes the position that evaluation is synonymous with the 
assessment of the absolute worth or folly of a particular project. Some- 
times referred to as the "summative" or "outcome" evaluation, this view 
limits the domain of the evaluation enterprise and makes it a totally judge- 
mental process which frequently engenders threat cind uncertainty in the minds 
of project staff. Historically, evaluators have tended to foster this con- 
ception, stressing the experimental design approach to evaluation which it was 
hoped would lead to the total and conclusive acceptcince or rejection of the 
endeavor. 

A program's worth cannot be judged by any set of data and accompanying 
statistics; people must still mcUce judgments of these data. Such judgments 
encompass the expectations of persons and their knowledge of the realities 
concerning the difficulty in administering programs, the stubborness of the 
problems they are frequently trying to solve, and their knowledge of available 
alternative solutions. Acceptance or rejection of a program cannot rest on 



the outcome of a single suxranative evaluation study any more them a scientific 
theory cam he proven or disproven on the basis of a single experiment. The 
total devotion of evaluation resources to the outcome issues represents a 
misplaced emphasis in our opinion. Instead, careful, self-conscious monitoring 
of the procedures (in5>lementation) and of the progress of the substemtive 
activities of the program (progress) must receive their share of the resources. 
In fact, it is our opinion that most service-oriented programs woxild better 
spend their limited evaluation funds in careful monitoring of the implemen- 
tation and formative phases of their programs rather than in the demonstration 
of the total appropriateness of the model under consideration. We see this 
latter matter as one for scientific study, the requirements of which can 
often not adequately be met within the service context or through the talents 
of the practitioners who are typically employed as program directors. Such 
endeavors should instead be underteUcen by researchers who should work in 
consort with practitioners to study the efficacy of various practices. 

Program Evaluation versus Staff Evaluation 

The second difference between our approach to evaluation emd other 
approaches concerns the distinction between evaluating programs and evalu- 
ating staff. Staff evaluation is undertaken to determine the quality of 
employee work and the employee's ability to function with otiiers for the 
purpose of determining employment continucmce, merit raises, and promotions. 
Staff evaluations are evaluations of specific individuals. Program evalua- 
tion, on the other hemd, is cm evaluation of a model, em approach to a 
problem, for which a staff is assembled euid assigned roles according to a 
designated plem. The purpose of the program evaluation is not to determine 
the value of the staff members so much as it is to determine the value of 
the program plem itself. 

A single example should clarify this distinction. Imagine that a 
teacher is hired to provide consulting services to regular classroom teachers 
in order that the regular classroom teacher be better able to teach reading 
to some of tlie retarded readers in her class. The consulting teacher, if 
her performancre is to be evaluated at all, should be responsible for per- 
forming the activities contained in her job description to the best of her 
eODility. If the children do not benefit from the role behaviors that she 
had been asked to assume, one logically should not hold her responsible but 
instead fault the model in which that role is prescribed. 

Though the conceptual distinction between staff and program evaluation 
is really quite clear, in practice the distinction frequently blurs. Ad- 
ministrators often would like to learn about the competencies of the par- 
ticular staff working on special projects (staff evaluation) in addition to 
the efficacy of the project's plan. Also, project personnel (laudably) often 
feel responsible for the outcome of the program, even if they are not intel- 
lectually committed to the specific role they are asked to assume. The 
consequence of blurring the distinction between staff and program evaluation 
is that persons tend to fear data collection because they fear being evaluated. 
In the context of a large program, there are many events over which an 
individual staff member has little control. If these events appear to reflect 
unfavorably upon him, he will obviously be reluctemt to welcome evaluators' 
scrutiny. 

i ' 
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Project evaluation frequently, then, places staff in a conflict situation? 
on the one hand they share in the desire to learn about the project's suc- 
cesses cmd failures in order that the project cam be improved, but, on the 
other hand, they do not wcmt their false starts cmd failures recorded, written 
up, smd distributed. Nor, of course, do they wamt to chcmce their real-life 
consequences, such as continued employment, to depend on proj'ict success. 

How can this conflict be resolved? Certainly project personnel, like 
any other regular line appointments, must be accountable for their work be- 
havior. We suggest that the distinction between staff evaluation and program 
evaluation be made < xtremely clear to all special project personnel by the 
administrator responsible for staff evaluation. In our experience, it is not 
enough simply to claim that staff evaluation is not occurring in the context 
of project evaluation. The fact appears to be that humans are extremely 
judgmental beings. Thus to claim that only the project model is being 
evaluated has a hollow ring to it. Therefore, the supervisory personnel 
should specify the domain and criteria of the staff evaluation so that it 
can be discriminated from the program evaluation. If an administrator is 
truly interested in evaluating a model, it is absolutely necessary that this 
distinction be made in order that the specific collection of people that 
comprise the staff will work together with the project director in collecting 
the information that he needs to evaluate the model. If the staff remains 
concerned that the very information that they are collecting to evaluate 
the model will also be used to evaluate their individual performance, the 
obvious result will be a reluctance to cooperate with the data collection 
andr perhaps in some cases, an outright attempt to sabotage the effort or 
distort the information. 

In summary, if one wishes to learn something adDout a program's function- 
ing duJ ^.he problems and successes that it experiences, one must remove the 
project stiiff from the conflict position of incriminating themselves by the 
evaluation evidence- they must help gather. By making clear the local rules 
which will apply to staff evaluation, as opposed to that information which 
will be used solely to evaluate the efficacy of the program in which the 
people are working, the administrator can develop rapport with the project 
staff and this will assure a greater probability that the information that 
he xeams about the program will be reliable cmd provide a sound basis for 
further progrcim planning. 

Some Attitudes toward Evaluation 

Two prevalent attitudes exist regarding program evaluation that will 
need to be dealt with by program administrators: one is that evaluation re- 
presents an invasion of privacy, the second is that evaluation is a worth- 
less enterprise that drains dollars away from service to children. 

Invasion of privacy . The feeling that privacy is invaded by evalua- 
tion procedures is, of course, related to the staff versus program evalua- 
tion distinction just considered. It must be recognized that some persons 
will not trust the distinction between staff emd program evaluation no matter 
how clearly the distinction is made. Sometimes this attitude is reflected 

1o 
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in extreme positions such as asserting that persons should not be evaluated, 
or that contract clauses or tenure prohibit the evaluation that is desired 
by the project, whether or not evaluation focuses on the program instead 
of the staff itself. At other times, one finds a certain staff member who 
simply deals in a "paranoid" fashion with any judgment. Though we are not 
using the word paremoid in the strictest psychiatric sense, its flavor does 
portray the strength of some persons' convictions against evaluation. For 
whatever reason, distrust of evaluation is not uncommon even in its extreme 
forms. It is likely, therefore, that any sizable project will have staff 
members who shun evaluation. If these staff menibers are in positions which 
require cooperation or, more so, their active assistance in order to complete 
the program evaluation, it is obvious that the evaluation will encounter 
serious difficulties. 

We know of no ways beyond those suggested above to deal with these dis- 
trustful attitudes. (There may even be some truth to a person's fears 
about evaluation, for they may have previously been victimized by the evalua- 
tion process. Then too, those who fear evaluation may, in fact, have some- 
thing to fear because of their own personal limitations.) The point is that 
persons with such attitudes should not be placed in special project positions 
where they are capeJale of inhibiting or restricting the gathering of infor- 
mation that is necessary to make important judgments about the program's 
functioning. An important consequence is that the selection of a site for a 
project cind the staffing of the project must take into account the necessary 
evaluation demands cdiead of time in order to avoid persons who will jeopardize 
the data base upon which judgments about the pragram must rest. It is much 
simpler to avoid dealing with persons who fear evaluation when evaluation is 
a necessary part of a program than it is to deal with their fears once the 
program is under way. 

Evaluation as worthless data gathering . Another cttitude about evalua- 
tion that inhibits the effective utilization of evaluation technology por- 
trays evaluation as being essentially a worthless enterprise. We like to 
believe that such attitudes are generally unfounded. When such attitudes 
are based on experience with unsuccessful evaluations, the fault likely lies 
with the inadequacy of the e valuator and not with evaluation methodology per se 
(although evaluation methodology must be recognized to be an infant science) . 
We enthusiastically agree that huge amounts of time and money have been 
literally wasted in collecting data that proves to be worthless. Many data 
are worthless not because they fail to provide clear-cut conclusions but be- 
cause they were never capable of testing the issue in the first place. The 
kinds of information that should be gathered to assess certain issues and the 
structure this information must possess are described later in this mcinual. 
Knowledge of these more technical aspects and the sensitivity with which they 
are employed often makes the difference between a worthless evaluation emd one 
which genuinely aids decision making. 

The summary dismissal of evaluation as worthless can mean one of two 
things: the person simply has not been exposed to an exemplary evaluation 
procedure or s/he is basically afraid of evaluation as discussed above and 
is choosing to derogate the evaluation process rather than to express personal 
concerns aJaout whether he will be judged unfairly or whether his own inadequacies 
will be uncovered. We find that administrators can readily distinguish the 



person who is basically concerned and threatened by the evaluation re- 
quirements from one who has genuine reservations about evaluation's worth. 
In the former case, we recommend not involving the person in a position 
critical to programs which require program evaluations as noted above. In 
the latter case, such a person can be a valuable asset to a program in that 
such a person will likely not sit silently by while "data gathering for 
data gathering's sake" goes on. Instead, if the evaluation is truly not 
serving its purpose, such persons are often very vocal and will help make 
the evaluation a genuine service to the decision-making process and, hence, 
ultimately to the children the program serves. 

To summarize, we believe that evaluation information can be gathered 
which can be useful for decision making but agree that many times program 
evaluation, inadequately handled, leads to tremendous wastes of time and 
resources. Persons with a genuine, rational scepticism about evaluations' 
worth can be a genuine asset, ihey are the type of people who insist that 
the data being gathered answer some question or fulfill some need or demand. 
They are also the kind of people who will further insist that the information 
being gathered be the simplest, most cost-effective, and efficient way of 
answering the evaluation question. These are the kind of content specialists 
that a competent evaluator likes to work with, because the more useful he can 
make the evaluation, the greater the support he can garner from the staff in 
gathering the information for making subsequent decisions. 

Evaluation as Accountability 

Do you want to act accountably or do you want to be accountable? This is 
a question of integrity: there is no policy or rule that a clever administra- 
tor cannot circumvent or bend to meet the needs of his own department, while 
such aplomb can be a virtue, in dealing with the accountability issue, it can 
lead to an avoidance of the quality controls legitimately desired by funding 
agencies. 

In our experience, we find that professionals react negatively to assess- 
ments of their accountability. Professionals are not effectively made 
accountable by external policy requirements or threats. If evaluation methodology 
IS utilized to coerce professionals to act in new ways which are termed 

accountable," the implicit assumption is that they have acted in some unaccount- 
able way m the past. The clear implication is that unaccountable people are 
irresponsible paople. it is no wonder that professionals have not warmly 
embraced the accountability dictum that has recently surfaced. 

The adver3e reaction to the notion of accountability, epitomized by re- 
quired program evaluations, must be effectively dealt with by program administrators. 
The danger is that some professionals will react against the implicit charge 
that they have previously acted in an inaccountable , irresponsible way by 
finding ways around the requirements of evaluation rather than to approach 
evaluation with an open desire to learn. We personally believe that the 
characteristics of most of the roles involved in public education require 
a great deal of personal responsibility and integrity. No external agent 
will ever be capable of affecting a greater commitment and output through 
the threat of evaluation than can be achieved by having the responsibility 
for quality work reside in the individual. 

10 
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YQu may or may not agree with this view. If you agree that account- 
ability cannot be achieved through the threat of evaluation , it is necessary 
to convey this belief to your staff. Professionals want to hear that 
eve0.uation will aid their own judgment processes to determine the best 
course for their future professional endeavors. Tliey do not want evaluation 
used as a xionitor of their personal worth as professionals or as a watchdog 
to determine whether they are being "responsible" on the job. The notion 
of accountability encompasses these negative aspects. 

We think the approach to evaluation advocated in this manual preserves 
the integrity of the professional by attributing to him the motive to seek 
self -corrective, objective feedback. We have found the following statements 
generally characteristic of educational professionals: 

1) Being asked to be accountable is a legitimate request of any pro- 
fessional; however, if it is implied that one would not be accountable were 
evaluation procedures not to be in effect, he will become angry and unsup- 
portive of the evaluation effort. 

2) Accountability is acceptcibly thought of as the professional re- 
porting to the taxpayer regarding the success or failure of certain of his 
efforts. Consumers have a right to know what they are buying. Educators, 
like any other professionals, have no right to cloak their services in such 
secrecy that the buying public cannot evaluate whether greater or less 
support of the endeavor is warranted. 

3) If an evaluation as designed will not provide immediately, or 
ultimately, an improvement in professional practice, then the evaluation 
activity should be discontinued. If an evaluation activity is required by 
a funding agent, then one logically should consider whether continued 
support from that agent is worth the costs of engaging in unproductive 
evaluation activity. Assuming that the dependence of the funding agent is 
great, as is usually the case, one will likely have to make some compromise 
between what one feels is necessary for the good of the project and what is 
being required by the funding agent. 

4) Attitudes against perfunctory evaluations should be voiced because 
of the money they waste. An administrator should seek talented evaluation 
persons who can both satisfy the dictates of grants and contracts while 
also providing the project with genuinely useful information. These goals 
eu^e not incompatible. 

In summary, viewing the role of the evaluator as providing accountadjility 
a quality control mechanism - is demeaning to the professional staff. We 
encourage a benign view of accountcdjility as simply the taxpayer's right 
to know the efficacy of the products that he is purchasing. Evaluation, 
in this context, is a professional activity in and of itself carried out in 
consort with other professionals, for the purpose of improving their own 
impact and for demonstrating their efforts to the "consumer". This view 
rejects the notion that funding agents have the right to use "accountability" 
to threaten, control, and demean professionals. We believe that it 
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is professionally responsible to resist these attempts to create an ad- 
versary rather than a cooperative relationship between evaluation personnel 
and staff personnel. In the end, one must make the simple decision whether 
evaluation is a function performed for a funding bureaucracy or whether 
it is a self-corrective process ultimately in the service of professional 
growth and client welfare. Any view of evaluation which becomes subservient 
to policy rather than to information seeking and change for the better must 
be challenged. 



A Way of Thinking About Evaluation 



We propose three kinds or aspects of evaluation termed "implementa- 
tion," "progress" and "outcome evaluation," Implementation addresses the 
host of questions concerned with whether the program is organized, adminis- 
trated, and operated in a manner either prescribed by or desired by the 
initiators. Progress ev£U.uation concerns the program's movement toward its 
objectives and the modifications which might improve its rate of progress. 
Outcome evaU^uation addresses the broad question, "To what degree did the 
project meet its objectives?" 

Implementation Evaluation 

Implementation assessment is typically not included in evaluations, 
predominantly for historical reasons. Evaluation has usually been viewed 
in terms of experimental designs which were oriented towards outcome ques- 
tions - that is, whether the program did or did not meet its objectives. 
Administrative and organizational issues were typically left to the experi- 
ence and, often, intuition of those charged with implementing programs. With 
the increase in the nuinber of new programs stimulated by federal cind state 
monies, many educators, psychologists, and other social science professionals 
were required to take on administrative responsibilities for which they 
lacked training and for which they might not possess the requisite coit5>etencies. 
Consequently, we believe it useful to view the administrative and organiza- 
tionatl issues in an open, rationaU. manner and to ask whether the structure 
esteODlished to meet the project objectives and the administrative pro- 
cedures utilized in that organizational structure ccin reasoncQDly be seen as 
a sound basis for reaching the stated objectives. 

The efficacy of a program's plan, its "model", ccin never be fairly 
tested unless the program is implemented successfully. For exair5)le, one 
can obviously not assess the effectiveness of newly developed training 
materials for perceptually impaired children if the materials reach the 
classroom late in the school year. One cannot fairly evaluate the efficacy 
of the itinerant teacher support program if the itinercint teacher is exces- 
sively absent or if the teachers to whom she is to provide services are 
cintagonistic . Rather than relegate the host of intricate problems involved 
with administration and organization to such popular rubricks as "red tape," 
"bureaucracy" and "personality clashes," careful assessment of our capabili- 
ties in these areas, pinpointing wherever possible the problems cind rationally 
seek their solution should be made. In an extensive nationwide study of 
numerous model learning disability centers, directed by the first author and 
Dr. Steven Reiss of the University of Illinois, Chicago, educational admin- 
istrators proved to be extremely interested in talking cdDout problems and 
seeking workeOile solutions. Furthermore, they evidenced cin extreme degree 
of savvy in solving problems and avoiding those same types of problems in the 
future. Inexperienced administrators encounter many of the same problems and 
often create others which could have either been solved or avoided through 
use of an appropriate, self-conscious monitoring system. 

is 
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Progress Evaluation 



What we auce calling progress evaluation has been labeled by others 
"formative," "process," "procedural" and other terms. We have selected 
the term "progress evaluation" in order to stress the ongoing nature of 
this type of evaluation and the feedback that it embodies- Progress 
evetLuation assures that the staff remains sensitive to the objectives for 
which they were hired by providing informational feedback on the inter- 
mediate steps taken in order to reach the final objectives. For excunple, 
if one is organizing an in-service program to increase the knowledge of 
regular classroom teachers edaout certain characteristics of children re- 
quiring special education, one shoxild obtain feedback from the teachers 
along the way to determine whether changes in the inservice program format 
are needed. Rather than wait until twelve monthly sessions have been 
held to find out that the program is not serving the teachers* interests 
or needs, a simple questionna- or discussions following the session could 
serve to refocus the program and make it more likely to meet its stated ob- 
jectives. In essence, progress evaluation constitutes feedback as to 
whether one's objectives are being attained. 

Outcome Evaluation 

Outcome eveU.uation has historically been considered synonymous with 
the concept of evaluation, that is, "summative evaluation," or "product 
evaluation." Like progress evaluation, outcome evaluation is substantive 
in nature, concerned with whether the project has actueU.ly accoit?)lished 
its objectives or, more realistically, the degree to which objectives have 
been accomplished. Outcome evaluation has been the major focus of attention 
for some time. However, failure to attend to the implementation or progress 
phases may jeopardize one's chance of obtaining suitable outcomes and in learn- 
ing why suitable outcomes were not obtained. Thus, while outcome is important, 
whenever disappointing results are obtained at this stage, little is often 
leaorned as to why success was so limited. 

A redistribution of evaluation resources into the implementation and 
progress eveU.uation aspects will allow a much cleaurer underst2mding of a 
project's strengths and weaknesses so that future projects can benefit 
from past experience. Reports shoving no significant differences between 
treated and untreated children, for exanqple, tell us nothing about why 
greater gains were not made by the children in whom greater time and effort 
was invested. Early, self-conscious attention to the ongoing process would 
have shown that progress was not being made, and that procedural corrections 
were necessary. 

The distribution of resources within evaluation to address implemen- 
tation, progress and outcome is a highly individual and critical set of 
decisions. A later section describes the kinds of questions typically 
asked by different persons for each of these three aspects of evaluation 
and goes into depth aJaout various considerations one might utilize in 
deciding how to apportion one's evaluation resources. At this point, 
understanding the distinctions between implementation evaluation, progress 
evaluation, cUirJ outcome evaluation will allow us to consider the actual steps 
in planning an evaluation. 
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Planning an Evaluation: Initial Considerations 



Evaluation plamning should begin very early. As soon as the program 
has been conceptualized, evaluation consultation will be useful. Because 
the purpose of evaluation is to provide feedback information for corrective 
modifications of a program, it is important for project initiators to pro- 
vide the project with monitoring capcibilities . If evaluation is thought 
of only as a "component" of the project or as "accountability," it can 
easily be put off until later stages. In our opinion this is a mistake 
because the evaluation for corrective feedback needs to be part of the 
fabric of the program, not something appended later. 

We stress the early consideration of evaluation planning because of 
yet another consideration reflecting the reality of program genesis wittiin 
large educational systems: It is seldom the case that the project initiator 
is the person who carries out the day to day fui*ctioning of the project. Many 
times, in fact, neither the project initiator nor any of his immediate plan- 
ning staff become involved in the project's day-to-day functioning. Project 
initiators have many ways of insuring quality programs, such as placing 
them with a trustworthy person cuid/or in a school district that has operated 
similar programs in the past, or by assigning one of his best project monitors 
to the program, and so forth. We believe that building an appropriate eval- 
uation into the program in the beginning and having it part of the total 
design is one of the best ways to insure a quality program. 

In addition to our position that evaluation should be part of the fabric 
of the proposal and that it is a mechanism by which project initiators can 
help increase the probability that the program will meet its objectives, 
there are a ntimber of other considerations which warrant early evaluation 
plamning. Some of these considerations are relevant to the content of the 
evaluation and to its methodology while others are relevant to carrying out 
an evaluation. 

Evaluation Content and Methodology; Initial Considerations 

Consider first the question of how the content of the evaluation and 
the methodology are to be determined. In the initial phase of evaluation 
planning, it is important to ask what information the program will need to 
produce in order to survive. Unless one accepts that the new program will 
have a short life span, one should ask very pointedly what will have to be 
shown to whom for program continuation. Obviously the answer to this question 
is not always simple and varies with the funding pattern underlying the 
program. Will a school board, for instance, be the one to decide whether the 
program is continued? Will it be a federal granting agency who must rule on 
renewals? Does one's department have money which could be used to support 
a worthy program should external funding be removed? Is the program a 
demonstration/replication oriented and, hence, planned to have a limited 
life span? If the latter is the case, then one needs to consider who the 
recipients of the successful portion of the program will be and how they 
will fund the new ideas in their districts. The point is that very early 
on, one has to consider where these future dollars will come from and. 
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consequently, what information it will be necessary to show future 
"buyers" in order to have them adopt the project. 

Because program continuation or adoption of program elements by other 
projects or districts is contingent upon one's ability to demonstrate pro- 
ject success in one way or another, it becomes necessary to plsui 2Ui evalua- 
tion in accord with the information demands of future consumers. Given our 
position that evaluation must be woven into the project's fiber in order to 
be genuinely a service to the program, capable of iit^roving it, euid able to 
demonstrate its capabilities, planning for continuance must be a part of 
the decision making process in the formative stages of the project planning* 
More sinply, programs have typically two or three years to prove their 
worth. A project must be extremely efficient, hard working, and adept at 
problem solving if it is ever to demonstrate its viability in such a short 
period of time. A proper implementation and progress evaluation cem help 
the progreun isolate problems before they become disabling so that solutions 
can keep the project both on its time line and out of serious complications. 
A properly planned outcome evaluation can serve to substcuitiate the case 
that the program does, in fact, have the characteristics that the project 
initiator had hoped for. 

The question comes down then to whom must one demonstrate the project's 
viability. It is at this point where the realization is most vivid that 
prograun evaluation has a very large audience. To use a marketing metaphor, 
projects have many potential consumers who require certain evidence that the 
program is worth purchasing. When the project initiator recogni25es just how 
many persons there are who are interested in the outcome of the project he 
is designing, the problem question, "What is there here to evaluate?" changes 
to "Of the host of issues to evaluate, which ones shall I select for evalua- 
tion?" 

Let us look then at who is interested in the outcomes of special educa- 
tion projects. Those interested in whether the project is conducted as con- 
tracted and in the outcomes of the project include the funding agency {and 
ultimately the agency's own source of funding, frequently smother brsuich of 
government), state education agencies, the project initiator himself and his 
staff, the project staff and project director in particular, often local 
education agency officials where the project is conducted, persons immediately 
concerned with the project including teachers and parents, persons or 
organizations interested in the project such as parent groups, professional 
groups, and other school districts who themselves might be seeking an effi- 
cient means of solving similar educational problems in their schools. 

It is not simply the number of persons and agencies interested in the 
project that generates so many possible issues to evaluate. Rather, it is 
the diversity of interests exhibited; each group or agency views the pro- 
ject from a slightly different perspective and consequently seeks different 
information about the project. As project initiator, one can certainly not 
afford to collect the data necessary to answer all of the questions that 
these many interest groups would ask. We shall consider the actual process 
of selecting evaluation questions in a later section of this memual. At 
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this point it is importemt to recognize that various interest groups want 
to know various things about the project and will accept different kinds 
of evaluation infoxrmation. 

A funding agency typically is interested in (1) whether a project 
is administered according to the contract, particularly the budgetary 
aspects, (2) whether the program proves to possess the qualities anticipated 
by the initiators, and (3) what new information the project provides. Pro- 
ject personnel, another group intimately interested in the evaluation, treat 
the task more personally in that they often believe their subjective im- 
pressions provide a wealth of detail and insight missed by the evaluator. 
Other times project staff also seek information that their own experience 
camnot afford them. For example, a resource room teacher on a project 
may want to assess objectively the impact of her prescriptive teaching 
in vaurious subjects. A project initiator might think of only evaluating 
the over-all effects of prescriptive teaching without bothering to examine 
the relative success with each type of problem area being treated. A 
parent involved in the project as a teacher's aide may be interested in 
still other dimensions. 

Not only do other persons or agencies pose different questions but 
they will also accept different kinds of evaluation data. A parent volun- 
teer might be satisfied with the project and suggest refunding based solely 
on the perception that some of the target children seemed more enthusiastic 
and happy in school. The school board, considering further expenditures 
for similar projects throughout the district, may want cost-effectiveness 
estimates or evidence that the project not only increases the target children's 
emotional well-being but also their academic skills. 

Thus, there are two questions that the project initiator should ask 
himself: (1) who is interested in the project and consequently in its 
evaluation?; and (2) what do these persons want to know about the project? 
These concerns can be phrased as more detailed questions that a project 
initiator should pose to himself in the very early stages of the project's 
inception; First, "Who will want information sibout the project and who will 
I want to tell about the project's activities?" Second, "What will I wemt 
to tell each person and what evidence will I need to prove my points?" 

Conducting an Evaluation; Initial Considerations 

Not only will the question chosen for evaluation and the evaluation 
methodology be dependent upon early plcinning, but so will the problems that 
one will encounter in carrying out an evaluation. Even in its earliest 
formative phases, a program has a previous history. The persons working on 
the idea have previous relationships with one another; the orientation 
of the project has likely evolved over a lengthy period and certainly has 
not sprung full-blown at the present time; there may even be a previous 
project upon which this new program is consciously being modeled; the pro- 
ject may be reflective of a more general philosophy prevailing within the 
state or other administrative unit. The implications for evaluation of a 
project's prehistory are widespread. 
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Founding father . First, consider the role of program genesis • Whose 
idea was the program? Or equally important, who thinks the program was 
his idea? What is this person or persons present role during the pro- 
gram's inception? If he is still involved, will she/he be able to relin- 
quish control as well as responsibility to the persons who must carry out 
the program? If not still actively involved, will the "founding fathers" 
attempt to influence the project's direction? If not formally recognized 
or informally appreciated, will these persons remain supportive of the 
progra:n or will they feel it was "stolen" from them? In essence, the cjuestion 
is, "Whose 'baby' is this, and what implications does 'ownership* have for 
carrying out the evaluation?" Specifically, will the persons who believe 
themselves responsible for the idea be willing to cooperate with the eval- 
uation plans that are being formulated now, or will they remove their 
support or even actively sabbotage the program's efforts? 

Competitors , Related to the "founding father" role is the fact that 
new programs do not serve totally new needs. New programs typically repre- 
sent new solutions to pre-existing needs thought to be inadequately met by 
previous programs or strategies. As such, persons involved with creating 
new programs often have a substantial commitment to the notion that the new 
program will be superior to existing services. By implication, then, 
persons providing existing services could view themselves as being displaced 
and/or feel as though others believe they have failed. For example, consider 
the reaction of reading specialists who must work side by side with a 
new leaunning disability staff involved in the remediation of reading diffi- 
culties. There exists considerable professional rivalry. Or consider the 
more general problem faced by a program for handicapped youngsters which 
reqpiires a "specialist" to accomplish work that the regular classroom teacher 
cannot accomplish. Without adequate communication, the regular classroom 
teacher and the specialist may feel themselves in competition, with the 
regular classroom teacher then feeling inadequate or resentful because of 
her apparent inability to educate one of her pupils. 

One should consequently ask who would be threatened by this new pro- 
gram, who is implicity being replaced or, by implication, being found in- 
adequate at providing for the needs that the new program addresses. Having 
identified those persons or groups, one must then consider the likelihood 
that they might create problems for this new program. How then might the 
new program be structured or placed so as to minimize these potentially 
disruptive influences? Alternatively, one could consider how some of these 
"displaced" persons or groups might become involved in this new program so 
that their potentially negative impact can be minimized or even eliminated. 

Lines of authority . A third area of potential problems in carrying 
out an evaluation, which needs to be considered in the early planning phases, 
is the formal and informal line of authority connecting all those persons 
who will have a role in administering or influencing the project. This 
area is obviously very complex and its handling will depend enormously on 
the interpersonal savvy of the project initiator. 

The formal lines of authority are an issue of public record but we 
all recognize that various informal systems of influence exist based upon 
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past professional relationships, personal relationships, inter-personal 
power, political influence, and so forth. It is important to consider 
specifically with regard to evaluation whether the evaluation plan that is 
being generated will be workcdsle within the given administrative structure. 
Will those persons having either informal or formal control over evaluative 
decision-making be supportive of the evaluation procedures being drafted? 

Specific Considerations 

The line of authority present in the project and the initiator's con- 
sequent judgment of whether the evaluation procedures being considered 
would be acceptsd^le within the total educational system spawn a host of 
specific considerations relating to the project site selection. Many pro- 
grams haw extreme interpersonal problems because the evaluation methodology 
originally adopted by the program initiators was simply not acceptable to 
the program staff nor to the on-site administration. Consequently, the 
program staff was unwilling to collect the necessary data. 

Site selection . Site selection is based on multiple considerations, 
only one of which is the cQsility to ccirry out the desired evaluation. Other 
considerations are very important: the site's geographic location, its 
representativeness for the program, its expressed desire to support the 
program, the capability of the staff, the site's need for services, or the 
felt responsibility to provide the site with special services, the confidence 
one has in the local administrators to carry out the program both faithfully 
and competently, and so forth - 

Amidst these other considerations, the evaluation requirements thrust 
upon a special project site are often overlooked. Overlooking the evaluation 
demands is a crucial mistake because the evaluation is important not only to 
continue the program and find other adopters but to increase the chances 
of program success. Hence, one must ask the following questions of potential 
program sites: 

1) Is the site receptive to the idea of a special program placed there? 

2) Is it cognizant of the administrative problems that inevitsQaly 
accon^any such programs? 

3) Are site personnel philosophically in agreement with the basic 
goals of the program? 

4) Is the site in need of the program's services or will the program 
be "in the way" or "just another special program"? 

5) Are there persons at this site sufficiently experienced in 
caurrying out special programs or will additional persons need to 
be hired? 

6) will the site recognize the importarice of the evaluation effort 
and will it be supportive? 
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7) Will the site invest itself in the program's continuation and, 
hence, be motivated to provide the necessary evidence in its 
argument for continued funding? 

8) Will the site be receptive to visitors seeking information about 
the project and generally accept their information-dissemination 
responsibility? 

9) Will it accept the degree of administrative control that the pro- 
ject initiator and his level of administration wish to retain? 
Problems can arise: one results from placing a project over which 
one »^ishes to have no further control in a site where the host is 
willing to receive the project but does not wish to have adminis- 
trative responsibilities, the other results from placing a project 
in a site where the local administration wishes to have almost total 
control but the initiator does not wish to relinquish total authority. 

10) Is the site representative of the domain of potential sites that 
one would wish to have adopt the program? Obviously, if the site 

is unrepresentative, potential adopters will be wary of the progrcun's 
demonstrated worth. 

11) Is the administrative staff at the site deemed capable of forming 
the desired relationship with the initiating agency? 

There are additional issues about site selection which will be con- 
sidered later, in conjunction with the issue of what specific evaluation 
questions one selects and whether the specific site in question will be re- 
ceptive to collecting data on these questions and more generally whether people 
at the site will be supportive of the evaluation effort. 

In summary, project initiators must be very sensitive to the pre- 
history of the project they are creating. They must recognize that persons 
besides themselves have vested interests in the project and may even consider 
it as "belonging" to them. Additionally, other persons may be made to feel 
like they have failed or that resources that potentially could have been devoted 
to their programs were diverted to others. In all of these cases, it is extreme- 
ly important for the project initiator to assess the roles these persons might 
play in the new project, specifically whether they will be supportive or 
antagonistic toward it. He should do whatever possible to gain support for 
the project either by involving important others or by placing the project in 
a site where potentially negative influences can be minimized. The danger is 
to ignore these real issues deriving from the project's pre-history and in 
so doing jeopardize the very viability of the project. Unless the project can 
be adequately implemented, the project model under consideration cam never 
receive adequate testing. The project initiator wants to be certain that the 
project has a fair chance of proving its worth so that rational decisions can 
be made about the future continuation of the program or about its adoption in 
other districts. Selecting a fertile site for the project and carefully estab- 
lishing clear lines of authority between the project initiator, project staff 
and other site staff, are two very important steps. In the latter case not 
only the distribution of responsibility but also delineation of authority and 
"chain of command" are essentia^^-for the adequate operation of the program. 

kb 
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What Issues Might Be Evaluated? 



Before settling on specific issues which will comprise the project's 
evaluation, it is well to consider the range of possible issues that could 
be evaluated and to dimensionalize the domain so that potentially impor- 
tamt or useful issues axe not overlooked. In a subsequent section of this 
mamual we will discuss those considerations you might use to choose the 
specific questions that you will address in the project's evaluation. 

Parenthetically, you will recognize that we have been using the terms 
"evaluation questions" and "evaluation issues" interchangeably and have not 
yet specified exactly what is maant by them. Many of you will also be 
familiar with evaluations which are orgamized in terms of major goals within 
which specific objectives are posited? the objectives and the associated 
activities then form the basis of the evaluation. Specifically, the 
objectives are evaluated in terms of the degree to which each is realized. 
A later section deals with how evaluation issues and questions regarding 
the prograun can be recast in behavioral terms and structured in funding 
requests as goals and objectives. At this stage, it is simpler to discuss 
what questions one would wish to ask of a program rather than to rework the 
questions into behaviorally stated objectives. 

Let us consider the various groups or agencies who will be interested 
in educational prograun evaluation in order to gain a feel for the typical 
concerns each of these groups and agencies possess. We shall take up in 
turn the three aspects of evaluation described earlier: implementation, 
progress, amd outcome. 



Implementation Evaluation 

Recall that the implementation evaluation concerns the monitoring of 
the administrative and orgamizational aspects of the program. Its purpose 
is to assure that an appropriate structure is established within the pro- 
gram between the program and the larger systems within which it is operating, 
amd that administrative procedures are adopted to carry out the substantive 
activities in the amount and degree specified in the project proposal. 
Surprisingly few persons are cognizant of the prdDlems that poor organization 
and inconsistent administrative policies can heap upon a program. At the 
same time, very few persons appear surprised when such problems occur. We 
recommend that care be devoted to specifying exactly how the program will 
be organized and administered. 

By orgamization we meam primarily a chain of command and role responsi- 
bilities which specify the duties of each person and the role relationships 
between persons at various stations within the project and their relation- 
ships to people outside the project. For example, a full-service special 
education intervention program might have a testing director through whom 
all requests for diagnostic testing would be directed and who would have 
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authority to meUce decisions regeurding the priority of testing. However, 
he might lack the authority to admit a child into the program's remedial 
section without a staff meeting involving other members of the program. 

While the topic of role responsibilities and relationships is so basic 
to program implementation, it is nevertheless the fact in our experience 
that such issues are left to "common sense" or to the "basic understanding" 
of the persons involved. The organizational structure is never physicedly 
recorded nor are the job descriptions or presumed relationships between the 
people recorded. We do not believe it necessary to straight- jacket programs 
in ways that hamper their effectiveness. At the same time it is our 
experience that more problems arise from misunderstcuiding as to what one's 
responsibilities and authorities are than from formal limitations based on 
written job descriptions. We therefore recommend that projects delineate 
their organizational structure and the roles of each of the persons and of 
the expected manner of interaction. Such a written document then provides 
the project evaluator with a means of assessing whether these specified 
roles and role relationships are being fulfilled and whether the deviations 
from these expectations are causing problems within the total system or 
whether the roles need modification. Feedback from such an assessment 
would allow for the modification of either a role, or its relationship 
with another role, or for the certification that the structure as designed 
was functioning adequately. These recorded role expectations could then 
be discussed by the staff so that mutual understanding is maximized. 

The administrative aspects of the implementation evaluation have more 
to do with operating procedures than with organizational structure. 
Administratively, one is concerned with how efficiently decisions get made 
and activities get accomplished and whether the degree of efficiency is 
sufficient to accomplish the objectives on the stated time-line and is 
sufficient to maintain staff morale. For example, are persons on the newly 
created project being paid on schedule? Are materials being ordered and 
received according to the time-line? Are decision-making milestones being 
passed at the rate appropriate to the time-line? For example, are parent 
contacts made at the time stated in the grant? Is the testing of children 
for selection into the program being accomplished at the appropriate time? 
Are diagnostic follow-ups being accomplished when specified? 

The implementation evaluation (consisting of the monitoring of the 
organizational and administrative aspects of the program) has not been 
particularly prominert in the minds of many of the consximers of evaluations, 
except when the system fails to function properly. Then, everyone wonders 
what is wrong, typically explaining the malfunction in terms of "bureaucracy," 
"red tape," "personality clashes," or in terms of the alleged incompetence 
of some project member (s) or person (s) with whom they must relate. 

We can list the persons to whom an implementation evaluation might be 
relevant; these same persons will be found to be interested in the progress 
and outcome evaluations as well. 



- 19 - 



Funding agents . The concerns of funding agents with implementation 
evaluation sure obvious, though, as noted cibove, seldom are their concerns 
expressed in grant or contract proposal guidelines. We believe it is the 
result of the youthfulness of administrative sciences and organizational 
psychology that evaluation guidelines do not contain detailed information 
regarding what we are here calling "implementation evaluation." 

A funding agent wants to assure that the program functions smoothly 
in order that the efficacy of the siibstantive ideas under consideration 
can be fairly assessed. Their second interest in such evaluation involves 
what we consider to be the proper use of the term "accountability" : Has 
the project organized and administered itself in a professional manner such 
that achievement of its objectives is at least feasible? If a project can 
be accountable/ then the monies spent at the very least should net an 
understanding of whether the particular idea has viability. 

It is, indeed, regrettable that administrative and organizational 
problems are presently seen as sources of embarrassment for programs and 
therefore are denied and/or carefully hidden from view. Relegated to the 
status of "gossip," these problems will never be further understood until 
we are willing to admit to the complexities and difficulties of administra- 
tive and organizational activities and seek solutions for these problems. 

With the spirit of inquiry, you should apply the following list of 
questions to one's program as part of its implementation evaluation. 

Organizational Questions . 

a) What are the duties (job descriptions) of persons to be employed 
by the project? 

b) What are the duties vis a vis the project of persons not employed 
by the project, e.g. administrative personnel who will implement 
the project in their local education agency? 

c) What is the organizational structure embodying all of these 
persons, i.e., the lines of authority for carrying out the 
various major activities of the project? 

d) More specifically, what are the relationships between these various 
roles described in the grant? 

e. Have the decision-making procedures been specified? Are they 
clear and consistent? 

Administrative Questions 

a) For each major- activity that the project must carry out, what are 
the administrative plans for accomplishing each activity? e.g. 
How will equipment be ordered? How will persons be paid? will 
payment to project people follow local or state salary schedules? 
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b) Have materials been ordered in time to be available when they 
are required? 

c) Are there adequate physical facilities for the program? 

d) Have all staff positions been filled? 

Many other questions must also be asked that relate to the specific 
characteristics of the project. 

Other persons interested in the implementation evaluation . Whereas the 
implementation evaluation comprises a monitoring of whether or not each of 
the organizational and administrative activities is carried out as planned, 
the mere existence of the plan represents a shared set of expectations among 
project personnel and those with whom the project will interact. As such, 
administrators who must be involved with the program, from the funding agent 
to those carrying out the program, will find the written implementation 
evaluation objectives most useful. It will serve to clarify their role 
vis ^ yj-s th« project and give them the shared expectations about each 
other's responsibilities. 

This same situation holds for project staff members whose role and 
role relationships with other members of the staff, and with significant 
others outside of the staff, will be sufficiently well defined so that 
the formal aspects of their relationships with others will be mutually 
shared. 

Staff members and administrators may find that upon initiating the 
project some of the roles and role relationships sure not adequate to 
meet the objectives. Naturally, changes should then be made in the organ- 
izational structure and administrative procedures in order to maximize 
the objectives. The purpose of the initial drafting of the implementation 
measure is that it makes all concerned more aware of how things are organized 
and administered and, hence, more conscious of problems in this domain either 
before they occur or as they are occurring. In our experience it is more 
difficult to solve problems that derive from implicit assumptions about 
shared expectations which may, in fact, not be shared than it is to solve 
problems which have been labeled and recognized by the persons involved. 
Using our previous example, it is much simpler to negotiate the issue of 
whether or not the psychologist has the authority to enroll children he 
has tested in the remedial program if there are written guidelines which 
can be used as a touch- stone. Without written guidelines, some subset of 
his decisions may be accepted if they are consistent with those of his 
supervisor while others may be contradicted by this person when not accept- 
eODle. In such a situation, the domain of the psychologist's authority is 
never clarified and the possibility of an interpersonal problem arising 
between him and the supervisory person is likely. 

Generally, most persons who are served by the grant, usually the 
children and their parents, are not particularly interested in program 
implementation unless the program is seriously malfunctioning. Consequently, 
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there is little initial input that one might expect from parents, teachers r 
school board members r or other community leadars regarding program implemen- 
tation. It is only when serious problems occur that these people hear the 
"gossip f'* emd then become concerned about this aspect of evaluation. 

Progress Evaluation 

Progress evaluation is essential to the project staff members who can 
use these data to alter their course for meeting the project objectives. 
However f the existence of such an evaluation is a comfort to administrators 
who can as a consequence know "how the program is going. " Being primarily 
concerned with issues internal to the project, progress evaluation derives 
very little direct input from persons outside the project, except to the 
extent that these external persons' needs and interests are represented 
in the objectives toward which the progress evaluation is focused. For 
example, parents of handicapped youngsters may be particularly interested 
in an in-service program for teachers in their local school. A progress 
evaluation could be established which evaluates the success of each of the 
in-service program sessions, obtaining participant feedback on their value, 
on their format, and perhaps even on the degree to which new skills were 
acquired. While parents will certainly be interested in the outcomes of 
the sessions as a whole, they will likely not be interested in the analysis 
of each in-service session even though such data may make future sessions 
more effective ■'n meeting the project's objectives. 

Progress evaluation questions are of the following kind; 

a) Are the substantive activities described in the program being 
carried out on the time-line established in the project? E.g., 
Has there been one in-service session for teachers each month 
as prescribed? E.g., Have the fifteen children been screened 
and assigned to the resource room within the time period as 
specified in the proposal? 

b) For some activities slightly more complicated questions must 
be asked; Are the activities being carried out in the degree 
and kind prescribed in the contract? E.g. Are four individually 
tailored lessons being given each child each week in the resource 
room? Here, the degree (four sessions) and the kind of activity 
(individually prescribed instruction) must be assessed rather 
than simply the occurrence or non-occurrence of some behavior 

or event. 

c) For each of the objectives, will the progress shown by the 
outcome evaluation be sufficient to accomplish the objective 
within the time frame established in the project proposal? 
E.g., Do project teachers' best judgments of their pupils' 
criterion referenced weekly testing indicate to them that the 
child is making progress or is some modification in the lesson 
planning going to be necessary for the child to achieve at the 
level stated in the project's objectives? E.g., After 
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assessing the present amount of interest shown by other districts 
in the program will the resources being expended and strategies 
being adopted be sufficient to encourage other districts to 
^dopt the program during the next school year as called for in 
the objectives? 

The kinds of information required to answer questions of this latter 
type are obviously highly specific to the project objectives. However, 
the general point is that information must be gathered and then judged as 
to whether or not the objectives will be met. The basic principle here is 
that information should be sought in order to assess progress rather than 
simply speculating or hoping that the objectives will be attained. 

Outcome Evaluation 

The outcone evaluation is the most complicated of the three types of 
evaluation in that it is of interest to so many different groups and agencies. 
Also, its complications derive from the fact that outcome evaluation is unique 
to each project, whereas implementation evaluation and some of the structural 
aspects of the progress evaluation are relatively consistent from one project 
to the next. The potential list of questions that could be asked as outcome 
objectives of a project is infinite; the task of the project initiator is to 
assess which have the highest priority, which might produce the mcst infor- 
mation for future decision-making, and which are within the budgetary 
limitations of the program. 

When choosing outcome objectives, it is well to consider the domain of 
persons interested in the project's outcome and the kinds of questions these 
people will ask. 

a) The present funding agent cmd other funding agencies concerned 
with similar programming will be particularly interested in whether 
or not the program demonstrates anything new or confirms something 
that was previously thought to be true. 

b) Funding agents also must be recongized to be self-serving to 
some degree, in that their own credibility as a funding agent 
is dependent upon their success in choosing programs wisely. 
Hence, they need information to demonstrate their progress in 
accomplishing the goals for which their administrative unit was 
estaolished. For example, an agency established to serve the 
needs of blind children would not long exist (hopefully) were none 
of its programs beneficial to the blind. 

In o\ir experience, funding agents typically are quite vague about their 
own outcome objectives. They give relatively little direction to project 
initiators so that initiators are frequently found to ask, "What do they 
want us to evaluate?" While it is judicious to be concerned with the inter- 
ests of the funding agent because an evaluation pleasing to the funding agent 
might increase one's probability of obtaining further funding, such concerns 
are often overdone. Project initiators know more about their project than do 
the funding agents and are thus in a better position to determine the charac- 
ter of the outcome objectives. We are not suggesting that the project 
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initiators be blind to the interests of their funding agent. Quite the 
contrary, we are suggesting that they ask what they might be able to 
demonstrate that would show funding agents that their programs are worthy 
of support. The answer lies not with the funding agent but with a careful 
analysis of the project itself. This issue becomes clearer at local levels 
where the project initiator will have mor ^ interaction with those persons 
controlling resources. There the initiator has a much better opportunity 
to assess the interests of the potential funding agents and consequently 
can learn what evaluation information he would need in order to insure a 
program's continuance. 

The following list of questions might be useful in ascertaining the 
domain of possible outcome objectives: 

a) What motivated the program; that is, what needs promoted the 
establishment of the program? 

b) What is the program attempting to accomplish? Are there some 
primary objectives and some secondary ones? 

c) Who provided impetus to the project and what information would 
they want to see collected to maintain their interest and support? 

d) What were the specific interest groups who had a role in stimulating 
this project and still have an interest in some particular aspect 

of the program? What information would they want to see derived 
from the project? 

e) Remember your own needs as project initiator: As someone with 
latitude in selecting the focus of special prograims , what would 
you like to see this program demonstrate? 

f ) Keeping in mind the range of potential consumers of the progrsim, 
what kinds of evidence would each want from you to persuade them 
to adopt the program in their district? This Scime question 
asked of yourself is often a good starting point: "What would 

I ask of someone else were they trying to convince me of this 
program's merits? 

In summary, outcome evaluation questions can be characterized most 
simply by the following single question: " Who wants to Imow what about the 
program and what information will they accept as evidenca? 
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Selecting Evaluation Questions 



From a host of possible questions, you must select those that will 
maximally serve your interests and those of potential project consumers. 
You must decide the amount of money to be devoted to the evaluation enter- 
prise, how the money will be apportioned between the implementation, 
progress, cmd outcome evaluations, and specifically what the content of 
each of these three evaluation endeavors will be, 

Preliminary Considerations 

In preparing to choose evaluation questions, four rules of thumb are 
suggested: (1) obtain an evaluator's advice, (2) seek information from 
relevant sources, (3) involve project site personnel, and (4) be rational. 
These considerations will be discussed in turn. 

Obtain an Evaluator's Advice 

A number of different issues are involved in the selection of an 
evaluator. During this formative phase, you need not have determined how 
the project will conduct its evaluation. Consequently, the evaluator who 
assists in making the decisions regaurding the evaluation content and 
methodology may not be the same person who •ventually carries out these 
tasks. It is, in fact, likely that this evaluator will not be the one 
hired by the project because those evaluation persons available for consul- 
tation are typically not also available for the more detailed evaluation 
work on specific projects. Consultation is frequently obtained from some- 
one employed by the same agency as the project initiator, someone usually 
associated with the division whose task it is to prepare grant requests. 
There are definite advantages in having the evaluator help in determining 
the evaluation's content and methodology and carry it out. It is also 
desireUjle to have the project director involved from the very beginning. 
The realities of grant preparation within public education make it unlikely 
that the project evaluator can be selected prior to funding. 

Some Guidelines . In obtaining a consultant evaluator to advise in 
grant preparation, the following guidelines may be helpful: 

a) Select an evaluator who may be availcible to be or to assist the 
project evaluator. 

b) Select an evaluator who has had experience with similar programs. 

c) Select an evaluator with whom you have had previous dealings or 
who comes highly recommended by someone you trust. 

d) Select an evaluator who will listen to the specifics of the 
progrcun rather than one who appears to kiKJW how to evaluate 

the project before its detail is even described to him. We caution 
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people to beware of purchasing "stock evaluations;" we highly 
recommend that prior to retaining an evaluator, you ask him to 
read over this manual and discuss his reaction to it with you. 
This discussion should allow you to determine whether or not 
you will have sufficient rapport for a good working relationship. 
Your goal is to find someone who will be sensitive to the imple- 
mentation and progress stages of evaluation rather than someone 
who views evaluation solely as the application of an experimental 
design to a service program, that is, one who views evaluation 
as synonymous with what is here termed "outcome evaluation." 
There is no reason why an evaluator should not explain to you why 
his approach is superior to the one recommended here. You should 
be able to assess whether this person is capable of giving you 
the advice you need. If you cannot understand his arguments as 
to why one approach is superior to another, you should be con- 
fident enough to question whether it is truly your lack of knowledge 
that creates your misunderstanding or whether, perhaps, you are 
being "snowed." We firmly believe that evaluators, like any other 
project staff, should contribute to the total program with the 
same kinds of checks on their activities as are imposed on other 
project personnel. In dealing with an evaluator unknown to you, 
ask to see previous reports he has contributed to and note 
specifically the role he played. Examine these reports to see if 
they have the elements you desire. 

e) The specific skills that the evaluator should have include knowledge 
of budgeting, specifically the cost of various tasks that would 
comprise an evaluation. He should know about administrative 
organization and job duties at least to the extent that he is 
willing to draft them into the implementation evaluation. He 
should have knowledge of experimental design, specifically with 
the application of experimental designs to similar kinds of 
service progreuns that you are constructing. He should have know- 
ledge of various instruments potentially useful for answering some 
of the outcome evaluation questions. He should be capable of 
constructing individually tailored rating scales and checklists 
capable of assessing progress towards various objectives in the 
program. He should have experience in writing evaluation sections 
of proposals, particularly experience in drafting questions in 
behavioral terms so that they are amenable to objective assessment. 

These criteria are extensive, to be sure, but it is best to know 
what services you are getting and how difficult the evaluation task can 
be rather than to assume that amyone can do it. For instance, you should 
not assume that any psychologist can adequately accomplish an evaluation. 
School psychologists cure trained primarily in child assessment testing, 
not specifically in project evaluation. Many psychologists can be capable 
evaluators if they acquire the additional skills necessary. But you must 
not assume that they are capable of evaluation simply by virtue of their 
psychological background and training. 
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Evaluator's role . An associated issue concerns the specific relation- 
ship of the evaluator to the project. Although the evaluator who advises you 
in the formative phases may not be assigned to the project itself when funded, 
he can advise you on whom to select and what this person's relationship to 
the project should be. 

There should be one person connected with the project who assumes 
primary responsibility for carrying out the evaluation. Typically, this 
person should be directly responsible to the project director. 

Three general types of evaluation roles might be considered. In one, 
the evaluator is a project staff member, usually havirg other duties along 
with his evaluation responsibilities. Here, the evaluator is intimately 
familiar with the project. If handled with tact and maturity, this relation- 
ship can result in a superb evaluation, particularly in the implementation 
and progress aspects. Having a project member be the evaluator can also 
serve the function of defusing the evaluation by assuring the fellow staff 
that it is a project assessment by the staff and not an evaluation of the 
individual staff members. A person integrated with the staff - working with 
them - can develop the rapport necessary to utilize scientific methodology 
to answer priority questions in as sensitive a manner as possible. 

The problem here is that projects can seldom afford a staff member 
solely for evaluation, thus evaluator s having other relevant skills need 
to be found. Frequently, personnel selection operates in the reverse 
manner: the persons having specific administrative and substemtive 
functions are first selected and then it is determined whether any of these 
persons would be capable of doing the evaluation. Typically, none of the 
persons are sufficiently capable but failure to budget for an evaluation 
person often results in the project director and the psychologist sharing 
this responsibility, though neither is actually capable of the task. 

A second step of evaluation role is sometimes referred to as a "third 
party" or "external" evaluation, in this arrangement, the project subcon- 
tracts the evaluation to some outside person or agency. The major advantage 
with this type of evaluation is its objectivity, but it can suffer from 
this very virtue by being insensitive to the details of actual project 
functioning. The strength of the third party evaluation is primarily in 
the outcome phase, although third party evaluators with sufficient budget 
knowledge and with prior experience with the project's content can certainly 
produce exemplary implementation and progress evaluation. However, they 
must spend considerable time interacting with the staff in an ongoing fashion 
in order to construct sensitive implementation and progress evaluations and 
to monitor adequately the information derived from these evaluation steps. 

The third type of evaluation has elements of both of the first two, 
being an evaluation designed by someone outside the project who nevertheless 
has considerable knowledge about the project. Typically, this type of rela- 
tionship is afforded by an evaluator provided by a state or county education 
agency. Such a person might be paid part-time by the project or have his 




services donated by the parent agency. He may bring to the project a wide 
range of experience evaluating projects of similar content. Under this 
type of arrangement, the evaluator typically uses the services of other 
personnel in data gathering, and, at times, in scoring. The major problem, 
with this method is that the evaluator may be cast in the role of the 
intrusive outsider or emmisary of project staff superiors. By making the 
role of the evaluator as clear as possible, these problems can be minimized. 

Choosing between these three types of evaluation models may depend 
heavily upon available funds and upon the allocation of these funds to the 
various aspects of evaluation: implementation, progress, and outcome. 
Smaller projects will likely have to accept whatever evaluation services 
are available from the parent agency or utilize whatever expertise is 
possessed by the staff. With more sizaJale funding, the decision revolves 
around the distribution of resources to implementation and progress evaluation 
rather than to outcome evaluation. If the project is in its early stages of 
development (the first year or two) , we suggest that you stress careful 
monitoring of project procedures (implementation) and the degree to which 
the project is progressing toward its objectives (progress evaluation) . In 
this case, an ongoing relationship with project staff is mandatory, dictating 
that the evaluatoi* either be a part-time staff member or in some other way 
possess a working relationship with the staff. If, on the other hand, the 
project is more mature and the evaluation issues surround only its ability 
to meet its objectives, a carefully instrumented and designed outcome evalua- 
tion would be stressed. In such cases an external evaluator is suitable, 
although even in this case the external evaluator must spend considerable 
time learning about the details of the project objectives and procedures, 
so that the instrumentation chosen to evaluate the objectives is maximally 
sensitive to tne project's impact. 

Obtain Information from Relevant Sources 

The second preliminary step in selecting evaluation cjuestions is to 
obtain as much information as possible from the sources interested in the 
project, e.g., the funding agent, other administrative personnel, potential 
staff members, potential project site personnel, interested professional 
groups, and others. By thinking ahead and asking yourself who will be 
interested in this project once it is funded and ongoing, you can determine 
whether or not you wish to include these persons or at least seek their opinions 
during these formative planning stages. A rule of thumb here is that if you 
desire or require the support of a person or agency once the program is funded 
and under way, you would do well to involve them in the initial decision 
making- Even if you already know that these persons' ideas may not be entirely 
consonant w/ith your own, it is important to recognize that if these persons 
have the potential to disrupt the project's functioning, they are much more 
likely to do so if they have been excluded from the project's formulation 
than if they have been included (even if their ideas are not heavily repre- 
sented in the final project) . 

We are restating our position that the interest of various groups, 
including the funding agent, be specifically addressed when selecting the 
evaluation questions. Rather than be obsessive about what the funding agents 




ERIC 



want, it is well to ask. Rather than wondering whether this program will 
meet the demamds being made by a parent group, ask them. Very possibly 
the program will not suit their demands; however, with some modification 
unimportant to you the project might be changed in a manner so as to gain 
their support. 

Though we recognize that projects are often hastily organized in order 
to meet submission deadlines, the involvement of significant others in 
the early stages represents more than good public relations: it can be 
both a way of enriching the concepts and methods involved in the project 
and also a me2Uis of obtaining important support which will enhance the 
project's ability to fulfill its objectives. 

We believe that, whenever possible, it is better to plan to avoid 
problems by involving significant others who have the potential to create 
difficulties for the program. Any administrator knows that problems can 
never be totally undone. For example, there are almost always lasting 
effects that result from interpersonnel difficulties. Just list for your-- 
self those persons who would wish to be involved in program plemning who 
would feel slighted if not involved and you will recognize immediately 
that this list of persons represents a serious threat to project success. 
You owe it to yourself and to your project staff to avoid the potentially 
negative impact of these persons or agencies. 

Involve Project Site Personnel 

A third issue concerns what other persons should be involved in selecting 
the evaluation issues. Some compromise is obviously necessary: involving 
all potentially interested persons totally is unmsmageable administratively, 
so frequently a procedure is necessary to obtain some lesser level of input 
such that the needs of many parties are met and their possibly valuable input 
is received. However, certain key persons need to be considered when selecting 
the specific evaluation issues. In addition to your evaluation consultant, 
persons at the potential site should be involved at this time. We have 
previously discussed some of the reasons for their inclusion: the necessity 
of the site being receptive to the program; their willingness to accept the 
additional administrative demands the project imposed; their administrative 
willingness and capability of collecting the evaluation data necessary to 
satisfy the evaluation requirements of the funding agent and the interests 
of the implementor and gnificant others; and then general support of the 
program, preferably with a commitment to the program and a sense of owner- 
ship derived from being involved in the planning stages. 

The local administrative head. There is no substitute for the knowledge 
of the specific persons at the potential project site gained from your personal 
interaction with them. With such knowledge, you can determine how much involve- 
ment these persons desire. Some local administrators prefer only to be kept 
informed so that they can adequately field questions coming from their staff 
or from parents. Other administrators require a more substantive commitment 
to any program operating within their jurisdiction. These styles typically 
reflect one's total management approach; the administrator who believes 
the support of his total staff is necessary for the success of special procrrams 
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will require much more involvement both for himself and his staff than will 
the administrator who exercises more centralized decision making. 

Your knowledge of the line of authority in the local site that you have 
selected will tell you which persons need to be involved. There is clearly 
a single person at the site, its head administrative officer, who obviously 
must be notified and invited to participate. 

We advise that you describe to this head administrative officer that 
the evaluation task involves selecting those aspects of the program which 
will be subject to analysis. He should be told that he might consider 
involving all personnel who have line authority over persons operating 
within the special project. It is important that they have smooth working 
relationships in order to implement the project's evaluation. Usually, 
these persons occupy such roles as testing coordinators or heads of special 
services, persons who might feel displaced or threatened by the new program. 

Other Administrators . In addition, it is important to be sure that you 
have not overlooked persons in positions of line authority intermediate to 
you and the head administrative officer at the site. For example, if you 
are employed at the state level and plan to place a small resource room pro- 
ject in a suburban school, it is important that the district level administra- 
tive officers are aware of this intention, supportive of it, and have had 
the opportunity to become involved if they wish. Though this is a rather 
obvious example, the picture becomes more complex when placing programs in 
large cities where an additional city level of school administration exists. 
The compromise, of course, always lies between the problems of involving too 
many persons so that decision making proceeds inefficiently versus dealing with 
the future problems that omitting significant others can create. As you 
certainly realize, education administrators are heavily overcommitted and 
do not necessarily seek to become involved in every special program in 
their purview. Offering a person a chance not to become involved may serve 
your purposes just as well as offering them involvement. You, of course, 
have to be prepared to accept their involvement, but often you can get 
persons to "sign off" a project so that you have "touched all the appropriate 
bases" without necessaurily increasing the difficulty of getting your job done. 
All we are suggesting is that in the early formative phases, the same admin- 
istrative attention be afforded that you would normally give to program 
administration itself, because we believe that the formative phases are 
critically important for establishing either a supportive or non-nutrient 
environment for the new project. Programs which develop out of shared needs, 
and are orgamized with the input of tacit approval of persons throughout 
the relevant administrative struct\ires, have a much greater chance of survival 
than do projects lacking this careful preparation. 

Substantive personnel . The personnel at the potential project site are 
inqportant to the decision msUcing in a number of areas. However, we are pri- 
marily focusing here on the conduct of evaluation. Their presence specifically 
in this regard is to assure that they understand that this special project 
carries with it not only extra services but also extra administrative demands 
and extra demands upon some of their regular personnel. Frequently, education 
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projects are not sufficiently funded to permit the employment of highly 
trained and experienced evaluators. Consequently, data collection and 
sometimes even its analysis must fall to on-site professionals whose 
primary duties and competencies lie in other areas, such as teaching or 
psychometrics- The local administrative officer and his testing and eval- 
uation staff need to be informed about the additional duties they will 
undertake so that as initiator you will know whether the chosen site is, 
in fact, a fertile ground for the project. It is better to find out during 
these planning phases that your chosen site is not receptive to the additional 
demands that are necessary for program monitoring, even though it might 
otherwise be an acceptable site. It is obviously a very easy matter either 
to select an alternative site at this time or even to negotiate with the 
initially chosen district what other compensation they might receive for 
these additional duties than to solve the problems attendant on a misunder- 
standing later on. 

Who controls the evaluation ? A related caution is offered; because the 
issue of evaluation is so emotionally loaded, it is important that you as 
project initiator be extremely clear with on-site officials as to who will 
control evaluation decision-making. They need to know how the evaluator will 
be related to the project administratively: is he to be under their control 
or will he remain under your control? By local sites taking on a project, 
they are obviously placing themselves in a position where their own compet- 
encies will be tested either implicitly or explicitly. Knowing who controls 
the information flowing from the evaluator is a very critical issue for 
them. Do you, in fact, plan to retain authority over the evaluation or 
will you turn it over to the local site as soon as a mutually satisfactory 
plan has been drafted? whatever your decision, it is mandatory, we believe, 
that these expectations be made explicit and shared by you and the project 
site head administrative officer and his key staff members. 

Be Rational 

One final preliminary step in selecting evaluation questions is the 
development of a proper attitude. Be rational; recognize those aspects of 
the evaluation issue selection that you can control and those which you 
cannot. We shall discuss immediately be''ow what kind of considerations 
you might wish to take into account. You should recognize that you, more 
than any other person, have control over the evaluation procedure because 
you are designing it. You should also recognize that you may not have total 
control and that you may need to give up some of your control to the site or 
to other administrative officers. Furthermore, in recognition of the fact 
that the project you are initiating does have a prehistory, you need also 
accept that you do not have a "free hand" in choosing the evaluation questions. 
Experienced administrators recognize that life is never so simple as this 
and so the selection of evaluation issues is just another example of how 
administrators must deal with multiple contingencies. All we are stressing 
in this final point regarding selection of the evaluation issues is that 
you deal with this topic with the same degree of rationality and reasoned 
judgement as with any other administrative decision. 
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Settling on an Evaluation Strategy 



The first step in selecting evaluation issues is to plan the scope of the 
evaluation. Do you wish to include an implementation, progress, and outcone 
evaluation? 



Some considerations are the following: 

a) If the project is a multiple year program, you night postpone 
outcome evaluation until the second or third years. 

b) If the project outcomes are necessary for continued support for 
a second year, obviously you must produce results very quickly? 
you will need a talented evaluator sensitive to the project's 
nuances to be able to show positive value of a program in a short 
time. 

c) Is your site amenable to the scrutiny nece'ssary for an inplononta- 
tion evaluation? If not, you might consider eliminating this 
aspect of evaluation, but we recommend that if the site is not 
open to such scrutiny, it night not be an adequate site for your 
project. Competent administrators have run programs for many 
years without formal, written implementation evaluations. Our 
guess, however, is that it would not be these administrators 

who would be adverse to such scrutiny <^o long as the monitoring 
were not disruptive or inordinately expensive. Rather, it is 
those who tcind to sweep problems under the rug and hope that 
they will go away who are most likely to reject the monitoring. 
In our minds f the institution of a special program is sufficiently 
difficult to require all of the positive features it can garner. 
Strong f confident administrators cire as critical to progrcun success 
as are the staff members concerned with its siabstantive activities. 
Consequently, we caution project implementors to select their sites 
very carefully and to insist on some level of implementation 
evaluation. This degree of implementation evaluation should satisfy 
the implementor that he can assess whether a project is adoquately 
operating. 

d) A progress evaluation is mandatory. Even an evaluation limited 
to the routine recording by the project director of the program's 
accomplishments as it works toward its objectives is superior to 
the unchecked flow of activity. A project director's subjective 
judgements recorded in writing may be a very adequate source of 
information. Its writing allows other persons to assess the validity 
of these perceptions from their own perspective. We think that it 

is a serious mistake to allow new programs to operate for a lengthy 
period without formal, written assessment on their progress. Again, 
this attitude springs not from any concern about the integrity of 
the professionals involved in the programs, but from knowing about 
the value of defining and dealing with problems through self-conscious 
evaluation. 



Apportioning Resources Within Evaluation 



f 
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Noting the Evaluation's Audience 



Who is interested? A list should be made of all persons or agencies 
with whom the evaluation report will be shared. This list likely includes 
the funding agency, the person to whom you as project initiator are respon- 
sible, the persons throughout the administrative system funding the project 
and those receiving it, and so forth. Others may include oneself, other 
project staff, teachers and perhaps even pupils and parents involved in the 
project, funding agents from whom additional future support might be sought 
(such as state legislatures and local school boards) , and school administra- 
tors responsible for program planning. 

The purpose of making this list of persons and agencies who may be 
interested in the project's evaluation is simply to assure that the eval- 
uation will satisfactorily answer the questions such a varied audience 
might pose. This exercise makes o le cognizant of the multiple interests 
served by an evaluation and, thought you might not be able to pose and 
answer all the questions of this heterogeneous audience, you can select 
for evaluation those issues considered most important. 

What would they want to know? For each of the persons or agencies 
listed in the preceding step, you should describe each of the questions 
you would expect them to pose of the project. Consider their implication 
questions, their progress questions, and their outcome questions. This may 
seem like a big job and it is I It is from this total domain of possible 
questions that you will select the evaluation questions for the project. 
It would clearly be too extravagant to plan an evaluation for all of hese 
questions from every source, so use your evaluation consultant to give 
estimates of the difficulty and expense of evaluation each question. 

Responding to I.Tunediate Pressures 

Immediate pressures also need to be considered in selecting the 
questions . 

a) Is there any one or any group concerned about the administration 
and organization of programs similar to the one you are initiating? 
If so, are you put under pressure to select evaluation issues 
pertinent to their interests? This event is unlikely in our 
experience — even the most severe organizational and administrative 
problems do not appear to be sources of upset, perhaps because 
such problems are so typical. Also as noted above, such problems 
are often relegated to catch-phrase explanations such as "that's 
the way bureaucracy is" or alternatively are explained in terms 

of a personality clash between two persons. In either case the 
problems are not dealt with adequately. Therefore, you will likely 
not feel any immediate pressure to produce an implementation eval- 
uation though we believe you should accon.plish one to assure the 
program's adequate functioning. 

b) Is there a certain aspect of the proposed program which stems from 
widely felt needs or from the interests of a specific group? For 
example, a local parent group may have promoted a project to 
utilize parent aides in the school and, hence the aide's impact would 
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be a central issue for progress and outcome evaluation. 

c) Are you under pressure to show specific changes in the target 
population within a given period of time? If such is the case, 
consideraOjle resources need to be devoted to isolating potential 
areas of chcuige and measuring them sensitively. 

d) Alternatively, you might be under immediate time pressure to 
produce an interim report within a few months of the project 
initiation. In which case, your evaluator should concentrate 
heavily on the implementation and progress aspects of evaluation 
in order that data can be produced at an earlier date than would 
be possible with an outcome evaluation. 

Responding to Priorities 

Priorities are frequently established by immediate pressures on the one 
hand and concerns about longer range planning cind one's own interests on the 
other hand. 

a) Is the progrcOT related to a general state or local government plan 
for educational services in such a way that certain specific aspects 
of the program need to be evaluated according to the priorities of 
this broader plan? 

b) Does your position as program initiator give you long range planning 
capaQ^ilities and, hence, do you have specific interests in the 
kind of information to be derived from this program? If so, you 

should be sure that your own interests are represented in the evaluation. 

c) Remember that program longevity is an important issue to keep in 
mind during evaluation planning? how will you support this program 
or its staff after the program period has elapsed? This is the 
issue discussed earlier concerning who will possibly adopt the 
program and the evidence of the program's value they will require. 

Considering the Costs 

Pressures and priorities must be measured against the cost of answering 
certain kinds of questions. Monitoring systems capable of exemplary imple- 
mentation evaluations are very cheap. Outcome evaluation, which frequently 
requires multiple testing of children both in the special project and in 
control groups, is considerably more expensive. You will have to depend on 
your evaluator to estimate costs, based on his best judgement of how each of 
the questions you have isolated might reasonably be answered. However, for 
purposes of discussion let us look at a few examples of evaluation questions 
amd what kind of costs might be involved in each. 

Evaluation costs, of course, are intimately linked to the methodology 
used to answer the questions posed. The fact is that answering questions 
is fraught with uncertainty and is expensive. Furthermore, there is no 
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guarantee that the expenditure of money will result in unequivocal answers 
to even the most central questions. After all, if knowledge were cheap, 
we already would have purchased answers to all our questions. 

From this stage of selecting evaluation questions to drafting the 
evaluation procedures in writing, you will become increasingly dependent 
on the skills of the evaluator. While doing an exemplary evaluation is 
certainly a difficult task requiring technical competence as well as 
creativity on the part of the evaluator, it is not a task that needs to 
be shrouded in mystery. Therefore, there is no reason why the evaluator 
should not be able to explain to you the basis for his chosen procedures 
and the reasons that some alternative methods (which you might have suggested) 
were not deemed adequate. Again, we caution that there is no reason why the 
evaluator should not be able to communicate his rationale. 

If you do not understand the evaluator, do not simply assume it is 
your fault. There is no reason why the rationale cannot be adequately 
explained, even if you are unfamiliar with the technical aspects of the 
methods. We are not just concerned cibout protecting project initiators 
from unscrupulous evaluators; more importantly, you should be able to 
understand the logic behind the evaluation because it is frequently the 
administrator who must "sell" the project to others- He, therefore, needs 
to understand what the evaluation information is saying and why it is 
evidence for or against certain assertions. The next section of the manual 
describes some of the more technical aspects of evaluation planning but in 
a manner that we think will be understandable to the interested administrator. 
Specifically, it provides a brief overview of evaluation designs so that 
you can have meaningful input into both the selection of evaluation questions 
and the answering methodology. In the subsequent section, we describe how 
the evaluation should be drafted, utilizing behavioral terminology so that 
the evaluation issues can be handled in an objective, scientific fashion. 
These two sections should provide you sufficient familiarity with some of 
the evaluator 's tools so that you can assist in the decision making. 
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Evaluation Designs 



This section of the paper explains in some detail the nature of thirteen 
evaluation designs, their good and bad points, their inherent costs, and some 
examples. Actually, the first several designs are implementation and progress 
evaluation designs, while the remaining ones are primarily used for outcome 
evaluations. At their best, designs B, C, D, E can produce a mass of data 
and can certainly offer more implementation and progress information than 
Type A, and can facilitate a preliminary (and a very necessary) look at progrcun 
effectiveness. One cannot, however, reach any valid outcome conclusions with- 
out more powerful evaluation designs (e.g.. Types F through M) . 



Type A; NO EVALUATION Carried Out on the Activity 

Omitting an evaluation is often an oversight - but more frequently, because 
the project director is not required to evaluate for renewed funding, s/he 
ignores this task. In other instances, no money was budgeted for evaluation. 
When an activity is not formally evaluated, there is no way to determine its 
effectiveness although project staff and other project observers usually 
utilize their own subjective impressions (Type D design) to evaluate program 
success. This type of design involves no budgeted costs to the project. 
However, it can become costly in that the program funds may be being wasted 
on unverified procedures which may cause the program problems in the future, 
perhaps even its survival. 

Type B; A LOG of What Happened 

A non-evaluation design, the log is an enumeration of events that 
occurred in the life of the program (e.g., the number of children screened, 
the n\imber of teachers attending in-service sessions). As such, it simply 
records the day-to-day activities of the program, and not necessarily any 
of its effects; that is, it can only show that the program was implemented. 
However, in some instances, as for exai!5)le when the log shows an increasing 
number of teachers consulting with a specialist, or a decrease in the number 
of referrals, program effects can be inferred. When making such inferences, 
one must bear in mind that such numerical changes do not imply benefit, 
and furthermore that they may be produced by numerous other plausible causes. 

The costs of maintaining a log of events include secretarial time for 
record keeping and a system for keeping track of program happenings, in 
a teacher in-service program, a log might include counting the number of 
participants per session, the amount of money spent for workshop consultants, 
and the money spent for release time for the participating teachers. Whereas, 
for a program concerned with identifying children with lesirning problems, 
the log would include the number of pupils screened, diagnosed, and placed 
in remedial programs, the number and kind of test materials used, the number 
of hours spent testing, etc. In the situation where a resource room program 
is set up, logs could be kept on the number of children seen each day, 
types of materials they used, and the number of children returned to their 
regular classrooms at the end of the semester. 



Simple Designs 





Type C; Frequent MONITORING of the Activity 



In a true tine-series design, there is a series of measurements at 
periodic intervals before, during, and after a given prograun. This proced- 
ure allows the program director to be better able to pinpoint at what 
point in time (zmd under what circumstemces) chemges occurred. Such 
methods are extremely beneficial in progress evaluation where program modi- 
fication for program improvement is the purpose. Basically, this procedure 
involves more frequent logging of events and woiUd involve the additional 
costs necessary for gathering data more often. 

With respect to a teacher in-service program, monitoring would involve 
counting the number of participants in previous in-service programs and in 
subsequent programs, as a basis for coinparison; one might also secure infor- 
mation on participant's reactions to previous in-service workshops, the 
present one, emd subsequent ones in an effort to compaure this program with 
others . 

For an identification program, one could determine the nuittoer of 
pupils screened and diagnosed previous to the present program versus 
during its operation. A simileu: procedure could be applied to the test 
materials used. 

Type D; SUBJECTIVE Impressions 

Everybody alive has biases, conscious and unconscious. Impressions 
of a given activity can never be totally objective. They axe always 
affected by one's feelings and relationship to the activity. Project 
staff members have a large stcUce in the success of their project (i.e., 
continued employment, increased salaries, increased local and regional 
exposure, more status and prestige, etc.) Consequently, a subjective 
evaluation of one's own program must be accepted cautiously and points to 
the need for evaduation designs such as Types G-M. 

For progress evaluation, it is both acceptaODle and desirable to secure 
testimonials and other impressions. These will probably provide some ten- 
tative information about where a program is succeeding and failing. But 
these then need to be verified using objective measurement and experimentally 
stronger designs (see more complex designs — below). 

The evaluation of any part of one's program is typically colored by 
subjective impressions. For example, if teachers seem to be enjoying the 
workshop programs, one tends to assume they eare benefiting from it. 
Because children are diagnosed as learning disabled and then assigned to 
a remedial, program, one has a tendency to believe they are being helped. 
It is assumed that pupils attending a resource room five times per week 
will receive more help than those children with similar problems who stay 
in the regulaur classroom. 

The cost of such subjective impressions is nil;their value is almost 
as little in the determination of definite program effects. Adequate 
evaluation is going to be costly, as will be appeurent in the subsequent designs. 
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TYPE E: POST-activity Measures on a TREATED Group 



A minimum of useful scientific information requires at least one formal 
comparison. With a Type E design, the post-activity measure results cannot 
be compared to anything. In this case, we are not able to determine 1) if 
the participants are better off than before the activity, or 2) if they would 
have arrived at this point without benefit of the activity. On the basis of 
a post-test alone, anything said about the effect of the treatment activity 
runs the double risk of being both worthless amd misleading. 

As an exasvgle, achievement post-scores on children in a resource room 
setting could be collected or the total number of hours spent by all children 
during the year could be calculated for the resource room. These will give 
you numbers with quantitative values, but they will be virtually useless 
because you cannot tell if these numbers are relatively "good" or "bad," 
"high" or "low." 

The costs of evaluation increase when pupil testing comes into play. 
Administration, scoring and score amalysis of the tests is time consuming 
and will cost money in salaries, data analysis, and report writing time. 
In the case of this evaluation design, the above costs will be unwisely 
spent because so little information is obtained for the money. 

Type F; PRE- and POST-activity Measures on a TREATED Group 

A before and after measurement on one or more variables can show whether 
and how much the treated group changed during the intervening period. But 
with this design we are still unable to say that the treatment caused the 
chamge as many other simultaneous events might be responsible. Unfortunately, 
this design is commonly employed. It can be useful as a preliminary determina- 
tion of program effectiveness but is not definitive. 

This commonly employed design typically involves test administration 
before and after some event, such as a teacher in-service program or a year- 
long remedial program. It is expected to show that the participants increased 
their skills in some ability area. Such a change may be found but it is wrong 
to conclude that the program caused this change, or that this change could 
be caused only by this program. 

Doubled administration of tests and data collection procedures are 
required with attendamt increases in cost. This design will give you more 
information than Type E but still will not be conclusive. 

Type G; PRE-, DURING, and POST-activity Measures on a TREATED Group 

Cautions similar to those for the Type F design are advised in this 
case, although one can plot the progress of the group during treatment 
using this design. Noting the co\irse of change can be extremely helpful 
as a progress evaluation, especially in locating specific effective efforts 
within one's program. Several intermediate measures will allow you to 
determine more precisely how the group changed during treatment and what 
might have caused these changes. 
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More Complex Designs 



In the following situations where two or more groups are being compared, 
it becomes important to utilize randomization. The TREATED and CONTROL 
[and later COMPARISON) groups need to begin "equal" on each measurable 
comparison for meaningful interpretation. For example, if »ome antecedent 
event were to effect scores, then both the treated and control group scores 
would be equally affected; the same holds true for the other plausible causes 
for score change. When there exists a comparison group, the effect of an 
activity can be seen in one's treated group relative to the comparison group 
as the scores of both groups, other things equal, will both be affected the 
same by regression and other effects. So, if the groups differ on the post- 
test, it is likely due to the treatment. 

Type H; PRE- and POST-activity Measures on TREATED and CONTROL Groups 

In this case, the treatment is compared to the mere passage of time. 
Theoretically, the control group is not receiving any treatment, but in 
actual practice, this is rarely the case. Typically, the control group is 
receiving the regular educational program, if this regular educational 
program is a viable alternative to the experimental treatment under study, 
it is probably best to label it a COMPARISON group, since it will be retained 
if it proves as or more effective than the experimental treatment program. 
This evaluation design, when randomization is utilized , is very powerful in 
determining whether or not the treatment is better than the control program. 
Without randomization, any post-test differences between groups could be 
caused by inherent differences in the groups' composition. 

Type I; PRE-, D URING, and POST-activity Measures on TREATED and CONTROL 
Group 

The advantages of intermediate measures during a program are discussed 
under the Type G design. Cautions as listed for the Type H design are also 
applicable in this case. 

Type J ; PRE- an d POST-activity Measures on TREATED and COMPARISON Groups 

Comparison groups are treatment conditions alternative to the experi- 
mental treatment (e.g., resource rooms versus self-contained classrooms 
for the moderately learning disabled) . Having a randomly assigned compar- 
ison group allows you to compare the effects of the experimental treatment 
with the effects of an alternative activity, of course, unless subjects 
are randomly assigned to the groups, differential effects obtained may be 
due to pre-existing differences between groups. The pretest data can alert 
you of initial differences but you are then faced with serious statistical 
and interpretive problems. 

A question that is left unanswered in this design is whether the mere 
passage of time could have produced similar changes in both groups of sub- 
jects. Designs of Type L and M provide controls to allow this possibility 
to be evaluated. 

i / 
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Type K: PRE-yDURING, and POST-activity Measures on TREATED and COMPARISON 



Groups 



See the cautions listed under the Type J design and the positive 
features noted under design Type G. 

Type L; PRE- and. PQST-activity Measures on TREATED^ CONTROL and COMPARISON 



Assuming randomization, the Type L design will allow determination of 
whether the experimental activity is better than no activity at all (mere 
passage of time) and whether it is better than some alternative treatment. 
However, it is still necessary to consider carefully the possibility of re- 
active effects of pretesting and possible differences of instrumentation. 
It would be desirable to have an additional unpretested control or comparison 
group to measure the reactive effects of the pretest. One should also make 
sure that no change in measuring or scoring procedures could have added to 
or caused any observed differential effect of treatment. This design is 
obviously strong. Given remdomization, it provides many opportunities to 
study change. 

Type M; PRE, DURING, POST-activity Measures on TREATED, CONTROL and COMPARISON 



This design provides additional information to that available in Type L. 
Cautions are suggested under Type L while the advantage of intermediate 
testing is discussed under Type G. 



Groups 



Groups 
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Drafting a Written Evaluation Plan 



At this point, the funding request is ready to be drafted. The project 
initiator has roughed out the general procedures of the project (often 
referred to as the model) , consulted with his colleagues at the potential 
project site, involved someone trained in evaluation, listed all those persons 
and agencies that are interested in the project and noted the type of questions 
each would hope the project would answer. He should then consider the eval- 
uation design and the type of data necesseury to answer these questions and, 
with the help of the evaluator, estimate the cost involved. He then selects 
those issues for evaluation which are most germane to the project, which serve 
the immediate pressures wherever possible, and yet which are consistent with 
long range priorities, particularly those that enhance the survival value of 
the project and which are economically feasible. The evaluation costs are 
entered into the budget. With these preparatory steps completed, drafting the 
evaluation is a simple matter. However, a few comments about writing the 
evaluation might be helpful. 

Goals versus Objectives 

It is frequently useful to distinguish between the broad goals of a 
project and its more targeted objectives. The distinction is more one of 
degree than of kind? goals are generally considered to be long range, 
idealized outcomes which give direction to one's project. The following 
are examples of goals: to train classroom teachers throughout the state 
to be able to educate the learning disabled child; to prevent learning 
disabilities through preschool intervention; to provide appropriate special 
services to all of the state's learning disabled youngsters. Objectives, 
on the other hand, are more immediate targets to be achieved during the 
course of the project. Present practice dictates that these objectives 
be worded in behavioral terms in order to facilitate the evaluation of 
whether or not the objectives were met. Up until this point we have con- 
sidered "evaluation issues" and "evaluation questions" without even 
introducing the term objectives. We should examine now how the notion of 
a behavioral objective relates to what we have been calling questions. 

Questions versus Objectives 

The difference between questions asked about the project and behavioral 
objectives is basically one of wording. As project initiator, for example, 
one may ask whether the itinerant teacher model will be able to provide 
enough support to the regular classroom teacher to enable a learning disabled 
child to gain one month in his reading skills each month that he participates 
in the project. This question can be stated as an objective simply through 
rewording: "To provide prescriptive teaching in reading to each learning 
disabled child in the project for at least four thirty-minute sessions per 
week in order to produce at least one month gain on a standard reading test 
for each nvonth that the child is enrolled in the project." It really does 
not matter whether the project initiator translates the questions into 
affirmatively stated objectives or whether he leaves them as questions. 
There is value in stating questions as objectives in that they become 
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more action-oriented than skeptical. Also, stating questions as objectives 
provides the opportunity to translate often subjectively and grossly expressed 
ideas into clear, objective, behavioral terms. The important point is not 
whether the outcome to be assessed are stated as questions or as objectives 
but whether they are expressed clearly in behavioral terms so that empirical 
data can be the arbiter. 

Behavioral Terms 

Objective, behavioral terms help reduce the ambiguity in determing if 
the objective was met. For example, "To help each learning disabled child 
in the project achieve success in reading" is not clearly nor entirely 
behaviorally stated; the criterion for "success" is not specified nor is the 
nature of the "help." At the project's end, there could be disagreement 
as to whether the child were "helped" or even if helped whether he achieved 
"success." A more adequate behavioral statement of this objective was 
provided above; "To provide prescriptive teaching in reading to each learning 
disabled child in the project for at least four thirty-minute sessions per 
week in order to produce at least one month gain on a standard reading test 
for each month that the child is enrolled in the project." This statement 
does not change any of the project's procedures; it only clarifies what is 
to be done, what is the expected result from the intervention, and how the 
effects will be measured and assessed. One can objectively assess (given 
that data are collected) whether each child did indeed receive the four 
weekly sessions and whether his gains were as expected. In short, an 
objective is well-stated if it specifies the conditions (behaviors) necessary 
to assess empirically the degree to which the objective was achieved. 

Drafting the Evaluation Section of a Funding Request 

When preparing a written evaluation plan for submission as part of a 
funding request, note the following suggestions. Goals should be described 
in the introductory sections of the request. Objectives should be stated 
in the evaluation section. We suggest that a separate portion of the eval- 
uation section of the proposal be devoted to the three aspects of evaluation: 
implementation, progress, and outcome. For each objective, the evaluation 
activities that would be engaged in would be stated immediately following 
each objective. For completeness, one should include in the activities the 
statistical treatment of the data to be used, if any. The following would 
be a skeltal example of the appearance of such an evaluation section. 

Sample program. A series of teacher inservice programs; four two-hour 
sessions, one each Wednesday afternoon during the month of September; to 
inform teachers about the district's learning disabilities services, the 
referral procedure, characteristics of learning disabled children, and an 
introduction to methods for identifying and remediating these children; 
session leaders will consist of district learning disability specialists. 

Implementation . Program Objectives: all district teachers in grades 
K-4 are notified of the series and provided financial incentives to attend; 
district L.D. specialists will prepare objectives for the series and indivi- 
dual sessions; facilities are scheduled and materials are secured. 




Evaluation Activities: 



1. Ask santple of teachers whether or not they have been notified of 
inservice sessions; 

2. Ask district financial officer whether or not financial incentives 
are being offered; 

3. Obtain copies of objectives and determine adequacy; 

4. Determine if all materials are available when required during the 
sessions. 

Progress . Program Objectives: participating teachers will attend all 
four sessions; objectives for the sessions will be obtained; participant 
feedback will be used to help determine the structure and content of subsequent 
sessions. 

Evaluation Activities: 

1. Keep records of attendance; 

2. Attend sessions to determine if "script" has been followed; 

3. Develop feedback mechanism (discussion or written questionnaire) 
and assess its impact on subsequent sessions. 

Outcome ■ Program Objectives; participants will be able to list the 
L.D. services available and relate how to refer children for these services; 
they will know the characteristics of L.D. children and utilize this know- 
ledge for referral purposes; they will be able to describe the methods for 
formally identifying and remediating L.D. children. 

Evaluation Activities: 

1. Develop a written instrument measuring the degree of knowledge the 
participants have concerning district services, L.D. characteris- 
tics, screening and diagnostic methods and remedial instruction; 

2. Administer this instrument immediately following the last in-service 
session and one month later and analyze the results; 

3. For an eight week period (four weeks during and four weeks following 
the inservice sessions) count the number of children referred for 
possible L.D. and count the number finally diagnosed as L.D. by 

the specialist; 

4. During the week after the last session, ask participating teachers 
to describe the referral procedure. Also ask this of non- 
participating teachers. 
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Transferring Responsibility to the Project Director 

Frequently, project initiators and other proposal writers do not remain 
involved in the day to day functioning of the project. More typically, a 
project director is hired to direct the day to day functioning. The project 
director then works within the administrative structure at the project site, 
typically with the district level coordinating the fiscal administration of 
the project, and then with school level handling the administration of the 
substantive activities. Sometimes the project director is drawn from the 
staff at the project site though, in our experience, it is more typical that 
the project director and a sizable portion of the special project staff 
represent additional employees at the site rather than reassigned employees - 

Transmitting the Evaluation Plan 

The problem then arises of transmitting the evaluation plan to the 
project director. Additionally, one must assure that the project director 
obtains suitable evaluation assistance to carry out the plan laid down by 
you and your evaluation consultant. If significant members from the project 
site have been involved in the evaluation planning as has been recommended, 
some of the difficulties will already have been avoided. Nevertheless, the 
task of acquainting the project director and the on-site evaluator with the 
planning remains. 

We recommend that a copy of this manual be read by the project director 
and the project evaluator with you as initiator noting those sections that 
you thought particularly germane to your planning. By using the manual as a 
communication device, red pencilling those areas that you would like the 
project director to note, you can communicate your own attitudes about 
evaluation to the project director in a rather efficient manner. In so 
doing, you will also be establishing a basis of communication between your- 
self and the project director so that the expectations regarding the 
evaluation procedures can more readily be shared. 

Additionally, we reconmiend that this manual be examined by the on-site 
evaluator so that the orientation used in constructing the program's eval- 
uation can be transmitted to him without excessive amounts of time spent in 
IMtttings with the director and evaluator. 

Selecting an On-site Evaluator 

A carefully organized evaluation plan may often permit a less well 
trained person in evaluation areas to carry out an evaluation which other- 
wise would have been beyond his level of competence. This is to say that 
if appropriate preplanning as described in this manual has occurred and a 
careful written statement of the evaluation plan were accomplished, then 
in many cases the project psychologist or the project director may be able 
to carry out the evaluation with a minimum of extra assistance. The need 
for a professional, highly trained evaluator will be decreased significantly 
when the following conditions hold true: 
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a) In general, if the evaluation description is highly detailed, it 
can be carried out like any other set of activities by a component 
project director or his/her designate. 

b) If the focus is on outcome evaluation and the assessment instru- 
ments were specifically designated in the proposal along with the 
design to be employed, then carrying out the evaluation is 
primarily an administrative procedure. Such Issues as the manner 
in which the data should be organized emd analyzed might require 
consultation from a research psychologist or an evaluation person 
employed by a parent agency — but the general plan will already 
have been laid down. 

c) If the focus is on an implementation evaluation and the specific 
aspects of the administrative and organizational characteristics 
to be evaluated are detailed in writing, the monitoring of the 
administrative and organizational activities becomes purely an 
administrative matter. (Progress evaluation is the roost highly 
individualized of the three aspects of evaluation and, as such, 
requires someone competent and confident in the evaluation area.) 

Evaluation is not difficult so much as it requires a particular sensi- 
tivity to the ongoing aspects of the program and an appreciation for the 
need to utilize objective information in answering questions. A person 
with confidence, basic training, and a commitment to this type of problem 
solving can typically accomplish this task. At present, we plan to construct 
a manual geared specifically to progress evaluation for use by a project 
director so that the on-going functions can be monitored without incurring 
the heavy expenses of evaluation consultants. However, lacking specific 
directions, project directors should not be delegated evaluation responsi- 
bilities unless evaluation consultation funds are provided or, of course, 
unless he or she has expertise in this area. 



Summary 



Had you, as project initiator, followed our comments and suggestions 
to this point, you would have spent considerable effort and time developing 
an evaluation plan for your program. During this preparation phase you 
would have: 

a) consulted with stater district and local educational administrators 
about their expectations for the program; 

b) discussed with the local project personnel their concerns about 
the evaluation; 

c) considered the demands for success by other interested parties, 
such as the funding agency, the public, and so forth; 

d) with your evaluation consultant, prioritized these success expec- 
tations, made them into specific program objectives, and decided 
upon evaluation procedures for each objective; 

e) made sure that the evaluation covered the area of implementation, 
progress, and outcome; and finally 

f ) transmitted this evaluation plan to the actual project director 
and the person on-site responsible for carrying out the evaluation. 

Your responsibility relative to the evaluation is not yet completed. 
You must now be willing to receive the evaluation results as feedback for 
possible modification of the program model. Because you have been so per- 
sonally involved in developing the model, reports of program problems may 
be discouraging- However, it is the feedback and subsequent program modifi- 
cation that makes evaluation so worthwhile. By listening carefully and 
learning from the broad range of data a well-conceived evaluation provides, 
one can surely increase the potential success of future programs, both 
through the gradual process of refinement and the more creative process of 
new program development. 
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